You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "fhueske (via GitHub)" <gi...@apache.org> on 2026/06/05 17:00:27 UTC
[PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
fhueske opened a new pull request, #28329:
URL: https://github.com/apache/flink/pull/28329
## What is the purpose of the change
Stateful keyed two-input operator backing LATERAL SNAPSHOT joins.
* Two phases (LOAD/JOIN) gated by an operator UnionListState; mixed-phase rescale collapses to LOAD. Flip triggers: build-side WM reaches loadCompletedTime, or a processing-time idle-timeout fallback.
* Probe records are buffered in LOAD and joined on flip. Build-side changes are buffered per-key and applied lazily in event-time order on the next per-key access once the build-side WM advanced, preserving atomic -U/+U visibility.
* Watermarks: build-side absorbed; probe-side held back in LOAD, forwarded in JOIN. NULL equi-keys filtered via JoinConditionWithNullFilters.
* State TTL via keyed processing-time timers with a post-flip grace window; rearms are amortized so effective time-to-eviction is in [1.0×, 1.5×] stateTtlMs.
## Brief change log
* add `LateralSnapshotJoinOperator` according to FLIP-579
## Verifying this change
* `LateralSnapshotJoinOperatorTest`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no (operator is not accessible yet)
- If yes, how is the feature documented? n/a
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Co-authored-by: Claude Code 2.1.148 (Opus 4.7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3570685165
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
Review Comment:
Probably we can fix this by calling `scheduleIdleFlipTimer` not from `open` but when we see `WatermarkStatus.IDLE` on the build side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "pnowojski (via GitHub)" <gi...@apache.org>.
pnowojski commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3428075770
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
Review Comment:
Just be aware that due to long recoveries, this can take quite a lot of time. So even relatively values like a couple of minutes might not reliably detect "build side is idle".
_probably_ you could add a heuristic that the timer doesn't start ticking until first record on the build side arrives. However that wouldn't work if you don't expect any records from the build side anymore.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
Review Comment:
Have you tested it if watermarks are propagated downstream?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
Review Comment:
Can you explain in some javadoc why and how you are using namespaces?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
Review Comment:
> his method runs in a NON-KEYED context
Why does it work like that? 🤔 Why don't you just register a timer with keyed context to transition to the correct phase?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
Review Comment:
Here and in many other places please don't use abbreviations, especially that some of them I can't figure out what they mean.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
Review Comment:
I have a feeling that this will not work properly firing timers is interrupted due to a checkpoint. `super.processWatermark1(mark);` can return without firing all timers, and as such you shouldn't store/reference `currentProbeSideWm`, but rather obtain the current watermark value dynamically from the timer service.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
Review Comment:
Iterating over the whole state for each recovery just to provide metrics is I think a rather bad idea. This will take too long in many cases and brakes our convention that metrics shouldn't affect performance/stability of the job.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
Review Comment:
nit: no-op comments (and in other places)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3491197646
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
Review Comment:
The unit tests assert the propagation of the probe-side WM during JOIN phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3554790359
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
+ buildTableState.put(change, currentCnt == null ? 1L : currentCnt + 1L);
+ } else {
+ // -D / -U
+ if (currentCnt == null || currentCnt <= 0L) {
+ numUnmatchedBuildRetractions.inc();
+ return;
+ }
Review Comment:
I think now that we're gating the changes-to-apply by watermark; it's a more explicit contract with the user/source that all the related changes show up until that WM.
One discrepancy though is that we keep late INSERTs but drop late DELETEs (I'm referring to [this](https://github.com/apache/flink/pull/28329#discussion_r3547523930) comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3453804571
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
Review Comment:
The order should be -U/-D before +U/+I (if the event-timestamps are equal).
That's how updates are encoded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3507384248
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
Review Comment:
That's how operators encode an update. They emit first a retraction (-U or -D) and then an accumulation (+U or +I).
But you're right, we don't need to explicitly sort the changes. The change messages will arrive in the correct order (assuming that they came from the same upstream task).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3494391231
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
Review Comment:
> The expectation is that whenever we received a build-side record, it won't take long until we receive a watermark.
But it completely depends on the input data and WM strategy, doesn't it?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
Review Comment:
Thanks, that's what I meant by
> If that's to prevent out-of-order changelog events (I guess)
and `buildChangeBuffer` is supposed to holds these two:
> an upstream operator generates an update: -(A, 1), +(A, 2)
\- until the watermark.
So _**after**_ the watermark, we can flush that buffer to `buildTableState` so that it can be joined with `probe-side record on A: +(A, B)`.
Or, if the probe-side record comes earlier, then we join it with `{(A, 1), 1}`
But in the current code, we're flushing the buffer regardless of the WM, so (3) is possible.
Or am I missing something?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
Review Comment:
My concern was that we are doing all the work as one atomic unit (timer) per key.
Now with the fixed-size chunks it is better.
But now I have the following concerns:
1. 128 records might still be a problem - especially if the downstream operators multiply the output
2. Additional state (offset); the operator has quite a lot of them already. Each carries its own memtable (even if carved from the same budget), compactor, etc.
3. Unnecessary complexity: Flink has "interruptible timers" feature exactly for this use case; and the change seems to be a re-implementation of it at the operator level
----
> But that would require a few things:
> 1) configuration of the probe-side event-time attribute index
🤔 Maybe we can use size+1 as a timestamp? (Or rather `(size + 1) - Long.MAX_VALUE` to trigger on whatever non-negative watermark)
> 2) MapState<Long, List> because multiple rows can have the same timestamp, and
Yes
> 3) many more timers to manage.
Yes, but is it a problem?
Asymptotically, that doesn't change the picture (we already store N records on state).
Empirically, this adds: ~8 bytes per record to store, 1 more data access, timers overhead.
So the trade-off seems to be:
current approach: throughput + less IO
vs
interruptible timers: simplicity + stable checkpoints + less memory
I think I'd pick the latter.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3483285170
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
Review Comment:
But can they detect that the **joined** record is late - are both timestamps propagated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3492830901
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
Review Comment:
The timer namespace names are just for readability.
The operator only registers a single event-time timer (always on ts=1) and one processing-time timer (TTL).
So, there is not really a reason to distinguish between them.
We can drop the timer names and go for `VoidNamespace` if you think that's better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3491083477
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
Review Comment:
The event-time attribute of the probe-side is maintained and remains aligned with the probe-side WMs.
The build-side event-time attribute is "materialized", i.e., converted into a regular timestamp.
In LOAD phase, a probe-side record that arrives late is buffered and later processed like any other probe-side records. Since we do not emit any watermarks during LOAD, it won't be late once emitted.
In JOIN phase, a late probe-side record is joined against the current (processing-time) version of the build-side table and forwarded. Since probe-side WMs are forwarded as they arrive, the late record would still be late (and could be dropped by downstream operators if needed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3491319323
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
Review Comment:
A common (if not the most common) use case for this join is a stream enrichment scenario.
The probe-side is a fact table that is streamed along a dimension table (the build-side) and joined against its PK. So each probe-side record joins against at most one build-side record.
In this case, the FK and PK columns of probe and build-side (which are referenced in the join predicate) would be used as partitioning keys. Hence, the build-side table would be partitioned on its PK such that there would be only one record in each `buildTableState` map entry.
Therefore, we would not need to break the iteration because there is just a single row per key.
If you are referring to a different scenario, could to explain it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3501005377
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
Review Comment:
I've updated the PR. Please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3500577424
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
Review Comment:
In the current implementation we gate the application of the buffer with `currentBuildSideWm > bufferedAt`.
`bufferedAt` is the build-side WM at the time when the first record was put into the buffer and `currentBuildSideWm` is the latest observed build-side WM.
So we use the WM to trigger application of the buffered changes. But there were still corner cases that would result in only applying the retraction. I've updated the PR with a change to only apply changes with ts < currentWm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3437217159
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
Review Comment:
Since we already guard `buildRowtimeIndex < 0` here and the field count is known at construction, would it be worth adding the symmetric upper bound too?
```java
if (buildRowtimeIndex >= rightType.toRowType().getFieldCount()) {
throw new IllegalArgumentException("buildRowtimeIndex out of bounds for build row");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3483339686
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
Review Comment:
Do you mean https://github.com/apache/flink/pull/28329#discussion_r3444210195 ?
But that's only one case?
In general, `buildChangeBuffer` is unbounded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3483339686
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
Review Comment:
Do you mean https://github.com/apache/flink/pull/28329#discussion_r3444210195 ?
But that's only one case; in general, `buildChangeBuffer` is unbounded.
Or am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3491263187
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
Review Comment:
we store the build-side changes in the `buildChangeBuffer` and join probe-side records only against `buildTableState`.
We only apply the changes buffered in `buildChangeBuffer` to `buildTableState` once the build-side WM advanced to ensure that both update changes are applied in the same batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3491382854
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
Review Comment:
After the flip, we only buffer the build-side changes. Assume the following scenario (on a single key A):
* current `buildTableState` for A: `{(A, 1), 1}`
* an upstream operator generates an update: `-(A, 1)`, `+(A, 2)`
* at the "same time" we receive a probe-side record on A: `+(A, B)`
* because we cannot guarantee the processing order of rows arriving on different input and because we always join against the "current" version of `buildTableState`, there are multiple possible scenarios:
1. probe row before both build changes, result: `+(A, B, A, 1)`
2. probe row after both build changes, result: `+(A, B, A, 2)`
3. probe row between both build changes, result is empty because retraction was applied, but insert wasn't.
The build-side buffer prevents scenario 3 and "converts" it into scenario 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3492612924
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
Review Comment:
I'm registering a single timer (on TS=1) per key with probe-side records to be able to trigger them by forwarding the last observed probe-side WM. The namespace is actually not really needed because the operator only registers one event-time timer per key on TS=1, so the namespace is not really needed to distinguish between timers. It's just an additional hint for the reader.
Or do namespaces have significant overhead?
I was thinking about timers on probe-side timestamps.
But that would require a few things: 1) configuration of the probe-side event-time attribute index, 2) MapState<Long, List<Row>> because multiple rows can have the same timestamp, and 3) many more timers to manage.
I've opted for a different approach for more fine-grained timer processing to ensure that the flip processing can be interrupted. I'm processing the probe-side records in chunks of (currently) 128 records. If the number of records is larger, the timer persists its offset, registers a new timer on TS=1, and exits.
That should keep the number of timers in the order of keys (and not records) and ensure interruptible timer processing.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28329:
URL: https://github.com/apache/flink/pull/28329#issuecomment-4633777454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1894182265e8ec8de0d0908afa5207628080912",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f1894182265e8ec8de0d0908afa5207628080912",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f1894182265e8ec8de0d0908afa5207628080912 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3504363387
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
Review Comment:
This concern should be resolved by the migration to MapState and using individual timers for the flip processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3437217142
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
Review Comment:
Tiny thing: `Phase` is `public` but I don't think anything outside needs it (planner doesn't, tests share the package) - maybe make it package-private?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
Review Comment:
Looks like `getInternalTimerService(...)` is already registered up at line 408, so this second call seems to be dead - could we drop it?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
Review Comment:
Since we already guard `buildRowtimeIndex < 0` here and the field count is known at construction, would it be worth adding the symmetric upper bound too?
```java
if (buildRowtimeIndex >= rightType.toRowType().getFieldCount()) {
throw new IllegalArgumentException("buildRowtimeIndex out of bounds for build row");
}
```
It's defensive since the index is planner-supplied, but it's cheap and would fail clearly instead of an AIOOBE deeper down in `compareBuildChanges`
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperatorTest.java:
##########
@@ -0,0 +1,1666 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.KeyedTwoInputStreamOperatorTestHarness;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.data.writer.BinaryRowWriter;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.Phase;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.runtime.util.RowDataHarnessAssertor;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.flink.types.RowKind;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_CHANGE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_TABLE_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.PROBE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.deleteRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.insertRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateAfterRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateBeforeRecord;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.within;
+
+/** Harness tests for {@link LateralSnapshotJoinOperator}. */
+class LateralSnapshotJoinOperatorTest {
+
+ // ----------------------------------------------------------------- Schema
+
+ /** Probe row schema: (id BIGINT, key VARCHAR, val VARCHAR). */
+ private static final InternalTypeInfo<RowData> PROBE_TYPE =
+ InternalTypeInfo.ofFields(
+ new BigIntType(), VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+
+ /** Build row schema: (key VARCHAR, val VARCHAR, rt BIGINT). */
+ private static final InternalTypeInfo<RowData> BUILD_TYPE =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE, VarCharType.STRING_TYPE, new BigIntType());
+
+ /** Joined output schema: probe ++ build = (id, pKey, pVal, bKey, bVal, bRt). */
+ private static final LogicalType[] OUTPUT_TYPES = {
+ new BigIntType(),
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType()
+ };
+
+ /** Probe key column index (key VARCHAR is at field 1). */
+ private static final int PROBE_KEY_IDX = 1;
+
+ /** Build key column index (key VARCHAR is at field 0). */
+ private static final int BUILD_KEY_IDX = 0;
+
+ /** Build row-time column index (rt BIGINT is at field 2). */
+ private static final int BUILD_RT_IDX = 2;
+
+ private static final InternalTypeInfo<RowData> KEY_TYPE =
+ InternalTypeInfo.ofFields(VarCharType.STRING_TYPE);
+
+ private static final KeySelector<RowData, RowData> PROBE_KEY_SELECTOR =
+ nullSafeStringKeySelector(PROBE_KEY_IDX);
+ private static final KeySelector<RowData, RowData> BUILD_KEY_SELECTOR =
+ nullSafeStringKeySelector(BUILD_KEY_IDX);
+
+ private static final RowDataHarnessAssertor JOINED_ASSERTOR =
+ new RowDataHarnessAssertor(OUTPUT_TYPES);
+
+ // ----------------------------------------------------------------- Join conditions
+
+ /** Trivial join condition that always matches (equality is enforced by partitioning). */
+ private static final String ALWAYS_TRUE_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionStub extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionStub(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) { return true; }\n"
+ + "}\n";
+
+ /**
+ * Join condition that only matches when the probe value (field 2) equals {@code "match"}. Used
+ * to verify that the codegen'd condition is actually invoked at join time.
+ */
+ private static final String MATCH_VAL_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionMatchVal extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionMatchVal(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) {\n"
+ + " if (in1.isNullAt(2)) { return false; }\n"
+ + " return \"match\".equals(in1.getString(2).toString());\n"
+ + " }\n"
+ + "}\n";
+
+ private static GeneratedJoinCondition newTrueCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionStub", ALWAYS_TRUE_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ private static GeneratedJoinCondition newMatchValCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionMatchVal", MATCH_VAL_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ // ----------------------------------------------------------------- Operator / harness
+ // factories
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ GeneratedJoinCondition joinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return new LateralSnapshotJoinOperator(
+ isLeftOuterJoin,
+ PROBE_TYPE,
+ BUILD_TYPE,
+ BUILD_RT_IDX,
+ joinCondition,
+ filterNullKeys,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return newOperator(
+ isLeftOuterJoin,
+ newTrueCondition(),
+ new boolean[] {true},
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData>
+ newHarness(LateralSnapshotJoinOperator op) throws Exception {
+ return new KeyedTwoInputStreamOperatorTestHarness<>(
+ op, PROBE_KEY_SELECTOR, BUILD_KEY_SELECTOR, KEY_TYPE);
+ }
+
+ private static KeySelector<RowData, RowData> nullSafeStringKeySelector(final int keyIdx) {
+ return value -> {
+ BinaryRowData ret = new BinaryRowData(1);
+ BinaryRowWriter writer = new BinaryRowWriter(ret);
+ if (value.isNullAt(keyIdx)) {
+ writer.setNullAt(0);
+ } else {
+ writer.writeString(0, value.getString(keyIdx));
+ }
+ writer.complete();
+ return ret;
+ };
+ }
+
+ // ----------------------------------------------------------------- LOAD phase
+
+ @Test
+ void loadPhaseBuildSideChangeProcessing() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // During LOAD, build-side changes are buffered and later applied in event-time order.
+ // -D for a never-inserted (key, value) pair is defensively ignored.
+ addBuildChange(h, deleteRecord("k1", "ghost", 5L));
+ // Two identical records (same key/val/row-time) → count(k1, v1, 20) = 2.
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ // Earlier row-time than v1, but arrives later.
+ addBuildChange(h, insertRecord("k1", "v2", 10L));
+
+ // Still LOAD: changes are buffered, nothing applied or emitted yet.
+ assertPhase(op, Phase.LOAD);
+ assertThat(h.getOutput()).isEmpty();
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(buildTableKeys(h)).isEmpty();
+
+ // Advance the build watermark (still below the flip point) and access k1 again. The
+ // access drains the buffered batch in event-time order before buffering the new change.
+ addBuildWm(h, 50L);
+ addBuildChange(h, insertRecord("k1", "v3", 30L));
+ // TODO: also add updateBefore and updateAfter changes, to ensure that these are handled
Review Comment:
Small one - this `// TODO: also add updateBefore and updateAfter` flags a LOAD-phase `-U`/`+U` gap, so it'd be nice to either add the case or drop the TODO before merge
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
Review Comment:
Could we pull this out into a shared `clearAllPerKeyState()` helper? It looks like the same per-key clear shows up around line 860 and in the TTL-clear path, so sharing it would keep them from drifting
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
Review Comment:
`loadCompletedTime` is a boxed `Long` but it's mandatory (we `checkNotNull` it below), so a primitive `long` would read a bit cleaner here - the boxed `Long` makes sense for the genuinely-nullable `loadCompletedIdleTimeoutMs` / `stateTtlMs`
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperatorTest.java:
##########
@@ -0,0 +1,1666 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.KeyedTwoInputStreamOperatorTestHarness;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.data.writer.BinaryRowWriter;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.Phase;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.runtime.util.RowDataHarnessAssertor;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.flink.types.RowKind;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_CHANGE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_TABLE_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.PROBE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.deleteRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.insertRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateAfterRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateBeforeRecord;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.within;
+
+/** Harness tests for {@link LateralSnapshotJoinOperator}. */
+class LateralSnapshotJoinOperatorTest {
+
+ // ----------------------------------------------------------------- Schema
+
+ /** Probe row schema: (id BIGINT, key VARCHAR, val VARCHAR). */
+ private static final InternalTypeInfo<RowData> PROBE_TYPE =
+ InternalTypeInfo.ofFields(
+ new BigIntType(), VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+
+ /** Build row schema: (key VARCHAR, val VARCHAR, rt BIGINT). */
+ private static final InternalTypeInfo<RowData> BUILD_TYPE =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE, VarCharType.STRING_TYPE, new BigIntType());
+
+ /** Joined output schema: probe ++ build = (id, pKey, pVal, bKey, bVal, bRt). */
+ private static final LogicalType[] OUTPUT_TYPES = {
+ new BigIntType(),
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType()
+ };
+
+ /** Probe key column index (key VARCHAR is at field 1). */
+ private static final int PROBE_KEY_IDX = 1;
+
+ /** Build key column index (key VARCHAR is at field 0). */
+ private static final int BUILD_KEY_IDX = 0;
+
+ /** Build row-time column index (rt BIGINT is at field 2). */
+ private static final int BUILD_RT_IDX = 2;
+
+ private static final InternalTypeInfo<RowData> KEY_TYPE =
+ InternalTypeInfo.ofFields(VarCharType.STRING_TYPE);
+
+ private static final KeySelector<RowData, RowData> PROBE_KEY_SELECTOR =
+ nullSafeStringKeySelector(PROBE_KEY_IDX);
+ private static final KeySelector<RowData, RowData> BUILD_KEY_SELECTOR =
+ nullSafeStringKeySelector(BUILD_KEY_IDX);
+
+ private static final RowDataHarnessAssertor JOINED_ASSERTOR =
+ new RowDataHarnessAssertor(OUTPUT_TYPES);
+
+ // ----------------------------------------------------------------- Join conditions
+
+ /** Trivial join condition that always matches (equality is enforced by partitioning). */
+ private static final String ALWAYS_TRUE_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionStub extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionStub(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) { return true; }\n"
+ + "}\n";
+
+ /**
+ * Join condition that only matches when the probe value (field 2) equals {@code "match"}. Used
+ * to verify that the codegen'd condition is actually invoked at join time.
+ */
+ private static final String MATCH_VAL_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionMatchVal extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionMatchVal(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) {\n"
+ + " if (in1.isNullAt(2)) { return false; }\n"
+ + " return \"match\".equals(in1.getString(2).toString());\n"
+ + " }\n"
+ + "}\n";
+
+ private static GeneratedJoinCondition newTrueCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionStub", ALWAYS_TRUE_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ private static GeneratedJoinCondition newMatchValCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionMatchVal", MATCH_VAL_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ // ----------------------------------------------------------------- Operator / harness
+ // factories
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ GeneratedJoinCondition joinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return new LateralSnapshotJoinOperator(
+ isLeftOuterJoin,
+ PROBE_TYPE,
+ BUILD_TYPE,
+ BUILD_RT_IDX,
+ joinCondition,
+ filterNullKeys,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return newOperator(
+ isLeftOuterJoin,
+ newTrueCondition(),
+ new boolean[] {true},
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData>
+ newHarness(LateralSnapshotJoinOperator op) throws Exception {
+ return new KeyedTwoInputStreamOperatorTestHarness<>(
+ op, PROBE_KEY_SELECTOR, BUILD_KEY_SELECTOR, KEY_TYPE);
+ }
+
+ private static KeySelector<RowData, RowData> nullSafeStringKeySelector(final int keyIdx) {
+ return value -> {
+ BinaryRowData ret = new BinaryRowData(1);
+ BinaryRowWriter writer = new BinaryRowWriter(ret);
+ if (value.isNullAt(keyIdx)) {
+ writer.setNullAt(0);
+ } else {
+ writer.writeString(0, value.getString(keyIdx));
+ }
+ writer.complete();
+ return ret;
+ };
+ }
+
+ // ----------------------------------------------------------------- LOAD phase
+
+ @Test
+ void loadPhaseBuildSideChangeProcessing() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // During LOAD, build-side changes are buffered and later applied in event-time order.
+ // -D for a never-inserted (key, value) pair is defensively ignored.
+ addBuildChange(h, deleteRecord("k1", "ghost", 5L));
+ // Two identical records (same key/val/row-time) → count(k1, v1, 20) = 2.
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ // Earlier row-time than v1, but arrives later.
+ addBuildChange(h, insertRecord("k1", "v2", 10L));
+
+ // Still LOAD: changes are buffered, nothing applied or emitted yet.
+ assertPhase(op, Phase.LOAD);
+ assertThat(h.getOutput()).isEmpty();
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(buildTableKeys(h)).isEmpty();
+
+ // Advance the build watermark (still below the flip point) and access k1 again. The
+ // access drains the buffered batch in event-time order before buffering the new change.
+ addBuildWm(h, 50L);
+ addBuildChange(h, insertRecord("k1", "v3", 30L));
+ // TODO: also add updateBefore and updateAfter changes, to ensure that these are handled
+ // correctly
+
+ assertPhase(op, Phase.LOAD);
+ // Applied in row-time order: -D(ghost)@5 (ignored), +I(v2)@10, +I(v1)@20 ×2.
+ assertThat(buildTableKeys(h)).containsExactly("k1");
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L));
+ // The triggering v3 change is now buffered, awaiting the next drain.
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(1);
+ }
+ }
+
+ @Test
+ void loadPhaseProbeSideInputProcessing() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+
+ addBuildChange(h, insertRecord("k1", "build1", 1L));
+ addBuildWm(h, 20L);
+ addProbeRecord(h, 1L, "k1", "probe-load-1");
+ addProbeRecord(h, 2L, "k1", "probe-load-2");
+ addProbeWm(h, 50L);
+
+ assertPhase(op, Phase.LOAD);
+ // No output (records buffered, watermarks held back).
+ assertThat(h.getOutput()).isEmpty();
+
+ assertThat(op.getCurrentProbeSideWm()).isEqualTo(50L);
+ assertThat(probeBufferKeys(h)).containsExactly("k1");
+ assertThat(probeBufferForKey(h, op, "k1")).hasSize(2);
+ // Build-side change was applied to the build table.
+ assertThat(bufferedChangesForKey(h, op, "k1")).isEmpty();
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("build1", 1L));
+ }
+ }
+
+ // ----------------------------------------------------------------- Flip / transition
+
+ @ParameterizedTest(name = "leftOuter={0}, wmFlip={1}")
+ @CsvSource({"true, true", "true, false", "false, true", "false, false"})
+ void flipDrainsProbeBufferAndJoins(boolean leftOuter, boolean wmFlip) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(leftOuter, 100L, 200L, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ // LOAD: build state and buffered probes (one without a matching build row).
+ // k1 is a multi-set: count(v1)=2, count(v2)=1; k2 is a single row.
+ addBuildChange(h, insertRecord("k1", "build-k1-v1", 1L));
+ addBuildChange(h, insertRecord("k1", "build-k1-v1", 1L));
+ addBuildChange(h, insertRecord("k1", "build-k1-v2", 1L));
+ addBuildChange(h, insertRecord("k2", "build-k2", 1L));
+ // LOAD: add probe-side records
+ addProbeRecord(h, 1L, "k1", "probe-1");
+ addProbeRecord(h, 2L, "k2", "probe-2");
+ addProbeRecord(h, 3L, "k2", "probe-3");
+ addProbeRecord(h, 4L, "k3", "probe-no-match");
+ addProbeWm(h, 80L);
+ assertPhase(op, Phase.LOAD);
+
+ // assert that probe-side buffer is filled
+ assertThat(probeBufferForKey(h, op, "k1")).hasSize(1);
+ assertThat(probeBufferForKey(h, op, "k2")).hasSize(2);
+ assertThat(probeBufferForKey(h, op, "k3")).hasSize(1);
+ // idle-timeout timer is armed while in LOAD
+ assertThat(op.isIdleFlipTimerActive()).isTrue();
+
+ // trigger flip from LOAD to JOIN
+ if (wmFlip) {
+ // build WM crosses loadCompletedTime.
+ addBuildWm(h, 100L);
+ } else {
+ // proc-time exceeds idle timeout
+ h.setProcessingTime(200);
+ }
+ assertPhase(op, Phase.JOIN);
+ // idle-timeout timer is removed on flip (canceled by a WM flip, fired by an idle flip)
+ assertThat(op.isIdleFlipTimerActive()).isFalse();
+
+ // probe k1 joins k1's multi-set (count-respecting: 2x v1 + 1x v2 = three rows);
+ // probe k2 (2 rows) joins a single k2 build row;
+ // probe k3 doesn't have a matching build row. INNER: no output, LEFT OUTER: null-padded
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 80L);
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v2", 1L),
+ row(2L, "k2", "probe-2", "k2", "build-k2", 1L),
+ row(3L, "k2", "probe-3", "k2", "build-k2", 1L),
+ row(4L, "k3", "probe-no-match", null, null, null));
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v2", 1L),
+ row(2L, "k2", "probe-2", "k2", "build-k2", 1L),
+ row(3L, "k2", "probe-3", "k2", "build-k2", 1L));
+ }
+ // Probe buffer drained on flip; build table preserved (k1 keeps multi-set counts).
+ assertThat(probeBufferKeys(h)).isEmpty();
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(
+ Map.of("build-k1-v1", 2L, "build-k1-v2", 1L));
+ assertThat(buildTableForKey(h, op, "k2"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("build-k2", 1L));
+ }
+ }
+
+ @Test
+ void idleTimerRearmsOnBuildWatermark() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 1000L, 100L, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(10);
+ h.open();
+ h.setProcessingTime(60);
+ // Build WM advances → re-arm.
+ addBuildWm(h, 10L);
+ // Original idle deadline was 10+100=110. Re-armed to 60+100=160.
+ h.setProcessingTime(159);
+ assertPhase(op, Phase.LOAD);
+ h.setProcessingTime(160);
+ assertPhase(op, Phase.JOIN);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void flipJoiningInvokesCodeGeneratedJoinCondition(boolean leftOuter) throws Exception {
+ LateralSnapshotJoinOperator op =
+ newOperator(
+ leftOuter, newMatchValCondition(), new boolean[] {true}, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addProbeRecord(h, 1L, "k1", "match");
+ addProbeRecord(h, 2L, "k1", "skip");
+ addProbeWm(h, 120L);
+ addBuildWm(h, 100L);
+
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 120L);
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "match", "k1", "v1", 1L),
+ row(2L, "k1", "skip", null, null, null));
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, "k1", "match", "k1", "v1", 1L));
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void flipJoiningCompositeEquiKeys(boolean leftOuter) throws Exception {
+ // Probe schema (kA VARCHAR, kB VARCHAR, val VARCHAR); build schema additionally carries a
+ // row-time attribute (kA VARCHAR, kB VARCHAR, val VARCHAR, rt BIGINT) at index 3.
+ InternalTypeInfo<RowData> probeType =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE, VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+ InternalTypeInfo<RowData> buildType =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType());
+ final int buildRowtimeIndex = 3;
+ // Compose the composite key as a BinaryRowData with both key fields.
+ KeySelector<RowData, RowData> selector =
+ value -> {
+ BinaryRowData ret = new BinaryRowData(2);
+ BinaryRowWriter writer = new BinaryRowWriter(ret);
+ if (value.isNullAt(0)) {
+ writer.setNullAt(0);
+ } else {
+ writer.writeString(0, value.getString(0));
+ }
+ if (value.isNullAt(1)) {
+ writer.setNullAt(1);
+ } else {
+ writer.writeString(1, value.getString(1));
+ }
+ writer.complete();
+ return ret;
+ };
+ InternalTypeInfo<RowData> compositeKeyType =
+ InternalTypeInfo.ofFields(VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+
+ LateralSnapshotJoinOperator op =
+ new LateralSnapshotJoinOperator(
+ leftOuter,
+ probeType,
+ buildType,
+ buildRowtimeIndex,
+ newTrueCondition(),
+ new boolean[] {true, true},
+ 100L,
+ null,
+ null);
+
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ new KeyedTwoInputStreamOperatorTestHarness<>(
+ op, selector, selector, compositeKeyType)) {
+ h.open();
+ // build rows: two identical rows for key a-1, a different row for key a-1, one row for
+ // key a-2.
+ addBuildChange(h, insertRecord("a", "1", "b-a-1-1", 1L));
+ addBuildChange(h, insertRecord("a", "1", "b-a-1-1", 1L));
+ addBuildChange(h, insertRecord("a", "1", "b-a-1-2", 1L));
+ addBuildChange(h, insertRecord("a", "2", "b-a-2-1", 1L));
+ // probes: matching composite, non-matching composite.
+ h.processElement1(insertRecord("a", "1", "p-a-1"));
+ h.processElement1(insertRecord("a", "9", "p-a-9"));
+ h.processElement1(insertRecord("b", "1", "p-b-1"));
+ h.processElement1(insertRecord("b", "9", "p-b-9"));
+ // flip to JOIN phase
+ addBuildWm(h, 100L);
+
+ LogicalType[] outTypes = {
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType()
+ };
+ RowDataHarnessAssertor compositeAssertor = new RowDataHarnessAssertor(outTypes);
+
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ compositeAssertor.shouldEmitAll(
+ h,
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-2", 1L),
+ compKeyRow("a", "9", "p-a-9", null, null, null, null),
+ compKeyRow("b", "1", "p-b-1", null, null, null, null),
+ compKeyRow("b", "9", "p-b-9", null, null, null, null));
+ } else {
+ compositeAssertor.shouldEmitAll(
+ h,
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-2", 1L));
+ }
+ }
+ }
+
+ // ----------------------------------------------------------------- JOIN phase
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void joinPhaseImmediateInnerJoin(boolean leftOuter) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(leftOuter, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // First probe WM
+ addProbeWm(h, 10L);
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 10L);
+
+ // First probe — joined immediately.
+ addProbeRecord(h, 1L, "k1", "probe-immediate-1");
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, "k1", "probe-immediate-1", "k1", "v1", 1L));
+
+ // Another probe WM
+ addProbeWm(h, 20L);
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 20L);
+
+ // Second probe for same key — joined immediately.
+ addProbeRecord(h, 2L, "k1", "probe-immediate-2");
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(h, row(2L, "k1", "probe-immediate-2", "k1", "v1", 1L));
+
+ // Probe for non-existent key — no output (INNER).
+ addProbeRecord(h, 3L, "k2", "probe-no-match");
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(3L, "k2", "probe-no-match", null, null, null));
+ } else {
+ assertThat(h.extractOutputStreamRecords()).isEmpty();
+ }
+
+ // one more probe WM
+ addProbeWm(h, 30L);
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 30L);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void joinPhaseBuildSideChangeApplication(boolean appliedByBuild) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // Two identical records for k1 (count 2) at row-time 1, buffered during LOAD.
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // Buffer four changes for k1
+ addBuildChange(h, deleteRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v2", 102L));
+ addBuildChange(h, updateBeforeRecord("k1", "v1", 1L));
+ addBuildChange(h, updateAfterRecord("k1", "v3", 103L));
+ // Buffer one change for k2.
+ addBuildChange(h, insertRecord("k2", "v1", 101L));
+
+ // assert number of buffered changes
+ assertThat(op.getCurrentBuildSideWm()).isEqualTo(100L);
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(bufferedAtWmFor(h, op, "k2")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k2")).hasSize(1);
+ assertThat(buildTableForKey(h, op, "k1")).isEqualTo(Map.of("v1", 2L));
+
+ // Probe record with no build WM advance - changes are not applied yet
+ addProbeRecord(h, 1L, "k1", "p-1");
+ JOINED_ASSERTOR.shouldEmitAll(
+ h, row(1L, "k1", "p-1", "k1", "v1", 1L), row(1L, "k1", "p-1", "k1", "v1", 1L));
+
+ // increment build-side WM
+ addBuildWm(h, 110L);
+
+ // assert that all changes are still buffered
+ assertThat(op.getCurrentBuildSideWm()).isEqualTo(110L);
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(bufferedAtWmFor(h, op, "k2")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k2")).hasSize(1);
+
+ // trigger application of k1 changes by build or probe-side input
+ if (!appliedByBuild) {
+ addBuildChange(h, insertRecord("k1", "v4", 111L));
+ // assert that changes have been applied and removed from buffer
+ // the triggering change is appended to the buffer
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(110L);
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(1);
+ assertThat(buildTableForKey(h, op, "k1")).isEqualTo(Map.of("v2", 1L, "v3", 1L));
+ // assert empty output
+ assertThat(h.getOutput()).isEmpty();
+ } else {
+ addProbeRecord(h, 2L, "k1", "p-2");
+ // assert that changes have been applied and removed from buffer
+ assertThat(bufferedAtWmFor(h, op, "k1")).isNull();
+ assertThat(bufferedChangesForKey(h, op, "k1")).isEmpty();
+ assertThat(buildTableForKey(h, op, "k1"))
+ .isEqualTo(Map.of("v2", 1L, "v3", 1L)); // /
+ // assert join results (v2 carries row-time 2, v3 carries row-time 1)
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(2L, "k1", "p-2", "k1", "v2", 102L),
+ row(2L, "k1", "p-2", "k1", "v3", 103L));
+ }
+
+ // assert that k2 change is still buffered
+ assertThat(bufferedAtWmFor(h, op, "k2")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k2")).hasSize(1);
+ // apply k2 change and join
+ addProbeRecord(h, 3L, "k2", "p-3");
+ assertThat(bufferedAtWmFor(h, op, "k2")).isNull();
+ assertThat(bufferedChangesForKey(h, op, "k2")).isEmpty();
+ assertThat(buildTableForKey(h, op, "k2")).isEqualTo(Map.of("v1", 1L));
+ // assert join results
+ JOINED_ASSERTOR.shouldEmitAll(h, row(3L, "k2", "p-3", "k2", "v1", 101L));
+ }
+ }
+
+ @Test
+ void joinPhaseWmForwardingLogic() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L); // flip
+ assertPhase(op, Phase.JOIN);
+
+ // Build-side WMs after flip are not forwarded.
+ addBuildWm(h, 200L);
+ assertThat(extractWatermarks(h.getOutput())).isEmpty();
+
+ // Probe-side WMs in JOIN are forwarded.
+ addProbeWm(h, 150L);
+ assertThat(extractWatermarks(h.getOutput())).containsExactly(new Watermark(150));
+ h.getOutput().clear();
+
+ // another build-side WM
+ addBuildWm(h, 300L);
+ assertThat(extractWatermarks(h.getOutput())).isEmpty();
+
+ addProbeWm(h, 250L);
+ assertThat(extractWatermarks(h.getOutput())).containsExactly(new Watermark(250));
+ }
+ }
+
+ /**
+ * For buffered build-side changes sharing the same row-time, retractions are applied before
+ * accumulations.
+ */
+ @Test
+ void joinPhaseAppliesRetractionsBeforeAccumulationsAtEqualRowTime() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // Drive into JOIN with empty build state for k1/k2.
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // Buffer accumulation-first at row-time 105 on absent rows:
+ // k1: +I then -D ; k2: +U then -U ; k3: +U then -U
+ addBuildChange(h, insertRecord("k1", "v1", 105L));
+ addBuildChange(h, deleteRecord("k1", "v1", 105L));
+ addBuildChange(h, updateAfterRecord("k2", "v1", 105L));
+ addBuildChange(h, updateBeforeRecord("k2", "v1", 105L));
+
+ // Advance build WM, then access each key to drain its buffer in event-time order.
+ addBuildWm(h, 200L);
+ addProbeRecord(h, 1L, "k1", "p1");
+ addProbeRecord(h, 2L, "k2", "p2");
+
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+ assertThat(buildTableForKey(h, op, "k2"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 105L),
+ row(2L, "k2", "p2", "k2", "v1", 105L));
+ }
+ }
+
+ // ----------------------------------------------------------------- NULL keys
+
+ @ParameterizedTest
+ @CsvSource({"true, true", "true, false", "false, true", "false, false"})
+ void joinRespectsNullKeysFilter(boolean leftOuter, boolean filterNullKey) throws Exception {
+ LateralSnapshotJoinOperator op =
+ newOperator(
+ leftOuter,
+ newTrueCondition(),
+ new boolean[] {filterNullKey},
+ 100L,
+ null,
+ null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord(null, "v_null", 1L));
+ addProbeRecord(h, 1L, null, "p_null");
+ addBuildWm(h, 100L);
+
+ // test joining during LOAD -> JOIN transition
+ if (filterNullKey) {
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, null, "p_null", null, null, null));
+ } else {
+ assertThat(h.getOutput()).isEmpty();
+ }
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, null, "p_null", null, "v_null", 1L));
+ }
+
+ // test joining in JOIN phase
+ addProbeRecord(h, 2L, null, "p_null");
+ if (filterNullKey) {
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(2L, null, "p_null", null, null, null));
+ } else {
+ assertThat(h.getOutput()).isEmpty();
+ }
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(2L, null, "p_null", null, "v_null", 1L));
+ }
+ }
+ }
+
+ // ----------------------------------------------------------------- Watermark status
+
+ @Test
+ void buildSideWmAndWmStatusForwarding() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // probe-side WM
+ addProbeWm(h, 70L);
+
+ // LOAD phase: build-side becomes idle then active again. WM is advanced.
+ h.processWatermarkStatus2(WatermarkStatus.IDLE);
+ h.processWatermarkStatus2(WatermarkStatus.ACTIVE);
+ addBuildWm(h, 50L);
+ // assert that no WMs or WM statuses are emitted in LOAD.
+ assertThat(h.getOutput()).isEmpty();
+
+ // Drive into JOIN
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // assert that only the probe-side WM was forwarded after the transition
+ assertThat(h.getOutput()).containsExactly(new Watermark(70L));
+ h.getOutput().clear();
+
+ // JOIN phase: build-side becomes idle then active again. WM is advanced
+ h.processWatermarkStatus2(WatermarkStatus.IDLE);
+ h.processWatermarkStatus2(WatermarkStatus.ACTIVE);
+ addBuildWm(h, 200L);
+
+ // No records, WMs, or WM statuses emitted in JOIN.
+ assertThat(h.getOutput()).isEmpty();
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void probeSideWmAndWmStatusForwarding(boolean probeIdleDuringLoad) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addProbeWm(h, 25L);
+ addProbeWm(h, 50L);
+ // Probe-side WM statuses received during LOAD.
+ h.processWatermarkStatus1(WatermarkStatus.IDLE);
+ if (!probeIdleDuringLoad) {
+ h.processWatermarkStatus1(WatermarkStatus.ACTIVE);
+ }
+ // Absorbed during LOAD — nothing emitted.
+ assertThat(extractWatermarks(h.getOutput())).isEmpty();
+ assertThat(extractWatermarkStatuses(h.getOutput())).isEmpty();
+
+ // Flip to JOIN phase — last probe WM emitted.
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ if (probeIdleDuringLoad) {
+ assertThat(extractWatermarkStatuses(h.getOutput()))
+ .containsExactly(WatermarkStatus.IDLE);
+ } else {
+ assertThat(extractWatermarkStatuses(h.getOutput())).isEmpty();
+ }
+ assertThat(extractWatermarks(h.getOutput())).containsExactly(new Watermark(50));
+ h.getOutput().clear();
+
+ // WMs and WM status updates received during JOIN are forwarded.
+ addProbeWm(h, 150L);
+ assertThat(h.getOutput()).containsExactly(new Watermark(150));
+ h.getOutput().clear();
+ // set probe-side to idle
+ h.processWatermarkStatus1(WatermarkStatus.IDLE);
+ assertThat(h.getOutput()).containsExactly(WatermarkStatus.IDLE);
+ h.getOutput().clear();
+ // set probe-side to active
+ h.processWatermarkStatus1(WatermarkStatus.ACTIVE);
+ assertThat(h.getOutput()).containsExactly(WatermarkStatus.ACTIVE);
+ h.getOutput().clear();
+ // emit another watermark
+ addProbeWm(h, 200L);
+ assertThat(h.getOutput()).containsExactly(new Watermark(200));
+ h.getOutput().clear();
+ }
+ }
+
+ // ----------------------------------------------------------------- State TTL
+
+ @Test
+ void stateTtlRefreshesOnAccessAndEvictsInactiveKeys() throws Exception {
+ // stateTtlMs = 100. Timers are registered at 1.5 × stateTtlMs, so the deadline for access
+ // at t=0 is 150.
+ LateralSnapshotJoinOperator op = newOperator(false, 50L, null, 100L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ addBuildChange(h, insertRecord("k3", "v1", 1L));
+ addBuildChange(h, insertRecord("k4", "v1", 1L));
+
+ // State is NOT evicted during LOAD even after the deadline passes (TTL fires are
+ // rescheduled past the LOAD phase).
+ h.setProcessingTime(200);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3", "k4");
+
+ // flip to JOIN
+ addBuildWm(h, 50L);
+ assertPhase(op, Phase.JOIN);
+
+ // Touch k1, k2, k3 to reset their TTL in JOIN; leave k4 alone.
+ h.setProcessingTime(275);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3", "k4");
+ addBuildChange(h, insertRecord("k1", "v2", 2L));
+ addBuildChange(h, insertRecord("k2", "v2", 2L));
+ addBuildChange(h, insertRecord("k3", "v2", 2L));
+
+ // k4 evicted: it wasn't accessed since proc-time (0) and we flipped to JOIN at (200)
+ h.setProcessingTime(350);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3");
+
+ // Update build state for k1 and k2; leave k3 alone
+ addBuildWm(h, 60L);
+ h.setProcessingTime(400);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3");
+ addBuildChange(h, insertRecord("k1", "v3", 3L));
+ addBuildChange(h, insertRecord("k2", "v3", 3L));
+
+ // k3 evicted, not accessed since (275)
+ addBuildWm(h, 70L);
+ h.setProcessingTime(475);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2");
+ // Access k1 from probe side to reset its TTL again; leave k2 alone
+ addProbeRecord(h, 1L, "k1", "p1");
+
+ // k2 evicted, not accessed since (400)
+ h.setProcessingTime(550);
+ assertThat(buildStateKeys(h)).containsExactly("k1");
+
+ // k1 finally evicted.
+ h.setProcessingTime(700);
+ assertThat(buildStateKeys(h)).isEmpty();
+
+ // The probe joined against the (k1) multi-set state at proc-time 475 — entries v1,
+ // v2, v3.
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v2", 2L),
+ row(1L, "k1", "p1", "k1", "v3", 3L));
+ }
+ }
+
+ @Test
+ void stateTtlClearsAllPerKeyState() throws Exception {
+ // stateTtlMs = 100. Timers are registered at 1.5 × stateTtlMs.
+ LateralSnapshotJoinOperator op = newOperator(false, 50L, null, 100L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ // Buffered during LOAD
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 50L); // flip to JOIN
+ assertPhase(op, Phase.JOIN);
+
+ // Buffer a change in JOIN that does NOT drain (no further WM advance / access). This
+ // populates the change buffer and the buffered-at tag while the build table keeps {v1}.
+ addBuildChange(h, insertRecord("k1", "v2"));
+ assertThat(buildTableForKey(h, op, "k1")).isEqualTo(Map.of("v1", 1L));
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(1);
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(50L);
+ assertThat(ttlExpiryFor(h, op, "k1")).isEqualTo(150L);
+
+ // Fire the TTL timer (well past the eviction deadline).
+ h.setProcessingTime(1000);
+
+ // Every per-key state object is cleared, not just the build table.
+ assertThat(buildTableForKey(h, op, "k1")).isEmpty();
+ assertThat(bufferedChangesForKey(h, op, "k1")).isEmpty();
+ assertThat(bufferedAtWmFor(h, op, "k1")).isNull();
+ assertThat(probeBufferForKey(h, op, "k1")).isEmpty();
+ assertThat(ttlExpiryFor(h, op, "k1")).isNull();
+ }
+ }
+
+ @Test
+ void stateTtlRestoreResetsFlipProcTime() throws Exception {
+ // stateTtlMs = 50. Build write at t=0 arms TTL at 75. Flip at t=0 → flipProcTime=0.
+ // Original grace ends at 0+50=50.
+ LateralSnapshotJoinOperator op1 = newOperator(false, 100L, null, 50L);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.setProcessingTime(0);
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertThat(op1.getFlipProcTime()).isEqualTo(0L);
+ state = h.snapshot(0L, 0L);
+ }
+
+ // Restart at t=30 — flipProcTime is re-anchored to 30; new grace ends at 30+50=80.
+ LateralSnapshotJoinOperator op2 = newOperator(false, 100L, null, 50L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.setProcessingTime(30);
+ h.initializeState(state);
+ h.open();
+ assertThat(op2.getFlipProcTime()).isEqualTo(30L);
+
+ // At t=75 the recovered TTL timer fires. Grace check: now=75 < flipProcTime+stateTtlMs
+ // = 80 → reschedule rather than evict.
+ h.setProcessingTime(75);
+
+ // k1 still present because the grace window was re-anchored.
+ assertThat(buildStateKeys(h)).containsExactly("k1");
+
+ // advance proc-time to 105 to evict state
+ h.setProcessingTime(105);
+ assertThat(buildStateKeys(h)).isEmpty();
+ }
+ }
+
+ // ----------------------------------------------------------------- Snapshot / restore
+
+ @Test
+ void restoreFromLoadPhaseSnapshot() throws Exception {
+ LateralSnapshotJoinOperator op1 = newOperator(false, 100L, null, null);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.open();
+ // Build-side multi-set with a duplicate count, plus a buffered
+ // probe.
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v2", 2L));
+ addBuildWm(h, 10L);
+ addBuildChange(h, insertRecord("k1", "v3", 3L));
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertPhase(op1, Phase.LOAD);
+ assertThat(probeBufferKeys(h)).containsExactly("k1");
+ state = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator op2 = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.initializeState(state);
+ h.open();
+ assertPhase(op2, Phase.LOAD);
+ // Buffered probe and build-table multi-set (with counts) preserved across restore.
+ assertThat(probeBufferKeys(h)).containsExactly("k1");
+ assertThat(buildTableForKey(h, op2, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L));
+ assertThat(bufferedChangesForKey(h, op2, "k1")).hasSize(1);
+ // Gauge is rebuilt from restored keyed state: one probe buffered for k1.
+ assertThat(op2.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(1L);
+
+ // Trigger flip; the buffered probe is joined post-restore, count-respecting against the
+ // restored multi-set (2× v1 + 1× v2 = three rows).
+ addProbeWm(h, 50L);
+ addBuildWm(h, 100L);
+ assertPhase(op2, Phase.JOIN);
+ assertThat(probeBufferKeys(h)).isEmpty();
+ // Draining the restored probe and build buffers brings both gauges back to 0.
+ assertThat(op2.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ assertThat(buildTableForKey(h, op2, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L, "v3", 1L));
+
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v2", 2L),
+ row(1L, "k1", "p1", "k1", "v3", 3L));
+ }
+ }
+
+ @Test
+ void restoreFromMixedPhaseSnapshot() throws Exception {
+ // Subtask A: drive into JOIN with a buffered change for k1.
+ LateralSnapshotJoinOperator opA = newOperator(false, 100L, null, null);
+ OperatorSubtaskState stateA;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(opA)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(opA, Phase.JOIN);
+ // Buffer a build-side change
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ stateA = h.snapshot(0L, 0L);
+ }
+
+ // Subtask B: stay in LOAD (no flip-triggering build WM).
+ LateralSnapshotJoinOperator opB = newOperator(false, 100L, null, null);
+ OperatorSubtaskState stateB;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(opB)) {
+ h.open();
+ addProbeRecord(h, 1L, "k2", "p1");
+ assertPhase(opB, Phase.LOAD);
+ stateB = h.snapshot(0L, 0L);
+ }
+
+ OperatorSubtaskState combined =
+ AbstractStreamOperatorTestHarness.repackageState(stateA, stateB);
+
+ // Restore the combined state — phase must be LOAD because some subtask was LOAD.
+ LateralSnapshotJoinOperator opC = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(opC)) {
+ h.initializeState(combined);
+ h.open();
+ assertPhase(opC, Phase.LOAD);
+
+ // assert state: A's build table ({v1:1}) plus A's buffered change and B's probe.
+ assertThat(buildTableKeys(h)).containsExactly("k1");
+ assertThat(buildTableForKey(h, opC, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+ assertThat(bufferedChangesForKey(h, opC, "k1")).hasSize(1);
+ assertThat(probeBufferKeys(h)).containsExactly("k2");
+
+ // During LOAD the current build WM is still MIN_VALUE, so the recovered buffer (tagged
+ // at the pre-restore WM) is not drained; the new change is appended to it.
+ addBuildChange(h, insertRecord("k1", "v2", 2L));
+ assertThat(bufferedChangesForKey(h, opC, "k1")).hasSize(2);
+ assertThat(buildTableForKey(h, opC, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+
+ // Trigger flip
+ addBuildWm(h, 100L);
+ assertPhase(opC, Phase.JOIN);
+
+ // Access k1: the buffered changes drain in event-time order, then the probe joins.
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(bufferedChangesForKey(h, opC, "k1")).isEmpty();
+ assertThat(buildTableForKey(h, opC, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L));
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v2", 2L));
+ }
+ }
+
+ @Test
+ void restoreFromJoinPhaseSnapshot() throws Exception {
+ LateralSnapshotJoinOperator op1 = newOperator(false, 100L, null, null);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op1, Phase.JOIN);
+ // Buffer a -U/+U pair for each key (tagged at bufferedAt = 100).
+ addBuildChange(h, updateBeforeRecord("k1", "v1", 1L));
+ addBuildChange(h, updateAfterRecord("k1", "v2", 101L));
+ addBuildChange(h, updateBeforeRecord("k2", "v1", 1L));
+ addBuildChange(h, updateAfterRecord("k2", "v2", 101L));
+ state = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator op2 = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.initializeState(state);
+ h.open();
+ assertPhase(op2, Phase.JOIN);
+ assertThat(op2.getCurrentBuildSideWm()).isEqualTo(Long.MIN_VALUE);
+ assertThat(bufferedChangesForKey(h, op2, "k1")).hasSize(2);
+ assertThat(bufferedChangesForKey(h, op2, "k2")).hasSize(2);
+ // Gauge is rebuilt from restored keyed state: four build-side changes buffered
+ // (two for k1, two for k2).
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(4L);
+
+ // k2: accessed while no build WM has arrived since restore → eager drain.
+ addProbeRecord(h, 1L, "k2", "p-k2");
+ assertThat(bufferedChangesForKey(h, op2, "k2")).isEmpty();
+ // Draining k2's two restored changes leaves k1's two still buffered.
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(2L);
+ assertThat(buildTableForKey(h, op2, "k2"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v2", 1L));
+
+ // k1: still buffered (eager drain of k2 left latestBuildSideWm at MIN_VALUE); advance
+ // the build WM, then access → normal WM-gated drain.
+ assertThat(bufferedChangesForKey(h, op2, "k1")).hasSize(2);
+ addBuildWm(h, 200L);
+ addProbeRecord(h, 2L, "k1", "p-k1");
+ assertThat(bufferedChangesForKey(h, op2, "k1")).isEmpty();
+ // All restored changes drained; gauge back to 0.
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ assertThat(buildTableForKey(h, op2, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v2", 1L));
+
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k2", "p-k2", "k2", "v2", 101L),
+ row(2L, "k1", "p-k1", "k1", "v2", 101L));
+ }
+ }
+
+ @Test
+ void restoreRearmsIdleFlipTimer() throws Exception {
+ LateralSnapshotJoinOperator op1 = newOperator(false, 1000L, 100L, null);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.setProcessingTime(50);
+ h.open();
+ assertPhase(op1, Phase.LOAD);
+ state = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator op2 = newOperator(false, 1000L, 100L, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.setProcessingTime(150);
+ h.initializeState(state);
+ h.open();
+ assertPhase(op2, Phase.LOAD);
+ // Re-armed at open()'s proc-time + idleTimeout = 150 + 100 = 250.
+ h.setProcessingTime(249);
+ assertPhase(op2, Phase.LOAD);
+ h.setProcessingTime(250);
+ assertPhase(op2, Phase.JOIN);
+ }
+ }
+
+ // ----------------------------------------------------------------- Metrics
+
+ @Test
+ void currentPhaseGaugeReflectsPhase() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // LOAD = ordinal 0.
+ assertThat(op.getCurrentPhaseGauge().getValue()).isEqualTo(0);
+
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+ // JOIN = ordinal 1.
+ assertThat(op.getCurrentPhaseGauge().getValue()).isEqualTo(1);
+ }
+ }
+
+ @Test
+ void probeBufferedGaugeTracksLoadBufferingAndDrainsOnFlip() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(1L);
+ addProbeRecord(h, 2L, "k2", "p2");
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(2L);
+ addProbeRecord(h, 3L, "k1", "p3");
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+
+ // Flip drains all buffered probes across keys.
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ }
+ }
+
+ @Test
+ void buildBufferedGaugeTracksJoinBufferingAndDrains() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // Flip to JOIN with an empty build state (no build records during LOAD).
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+
+ // JOIN-phase build changes are buffered (their tag equals the current build WM).
+ addBuildChange(h, insertRecord("k1", "v2", 101L));
+ addBuildChange(h, insertRecord("k1", "v3", 101L));
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(2L);
+ addBuildChange(h, insertRecord("k2", "v1", 102L));
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+
+ // Advance build WM, then drain each key by accessing it from the probe side.
+ addBuildWm(h, 110L);
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(1L);
+ addProbeRecord(h, 2L, "k2", "p2");
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ }
+ }
+
+ @Test
+ void watermarkGaugesTrackBuildAndProbeWatermarks() throws Exception {
+ // High loadCompletedTime so the operator stays in LOAD for the first build WMs.
+ LateralSnapshotJoinOperator op = newOperator(false, 1000L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(Long.MIN_VALUE);
+ assertThat(op.getCurrentProbeSideWatermarkGauge().getValue()).isEqualTo(Long.MIN_VALUE);
+
+ // LOAD phase: both watermarks are tracked even though nothing is forwarded.
+ addBuildWm(h, 50L);
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(50L);
+ addProbeWm(h, 70L);
+ assertThat(op.getCurrentProbeSideWatermarkGauge().getValue()).isEqualTo(70L);
+ assertPhase(op, Phase.LOAD);
+
+ // Flip to JOIN.
+ addBuildWm(h, 1000L);
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(1000L);
+
+ // JOIN phase: probe WM gauge keeps tracking (guards the dedicated currentProbeWm
+ // field).
+ addProbeWm(h, 1500L);
+ assertThat(op.getCurrentProbeSideWatermarkGauge().getValue()).isEqualTo(1500L);
+ addBuildWm(h, 1100L);
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(1100L);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void fanOutGaugesTrackMaxAndAverage(boolean leftOuter) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(leftOuter, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // k1 multi-set: v1 x2, v2 x1 → fan-out 3; k2 single row → fan-out 1.
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v2", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ // a probe record during load
+ addProbeRecord(h, 1L, "k2", "p1"); // fan-out 1
+ // a probe record that does not match
+ addProbeRecord(h, 2L, "k3", "p2"); // fan-out INNER: 0, LEFT OUTER: 1
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // check merge stats after transition
+ assertThat(op.getMaxJoinFanOutGauge().getValue()).isEqualTo(1L);
+ if (leftOuter) {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isEqualTo(2d / 2);
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(0L);
+ } else {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(1d / 2, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(1L);
+ }
+
+ // join probe in JOIN phase
+ addProbeRecord(h, 3L, "k1", "p3"); // fan-out 3
+ assertThat(op.getMaxJoinFanOutGauge().getValue()).isEqualTo(3L);
+ if (leftOuter) {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isEqualTo(5d / 3, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(0L);
+ } else {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(4d / 3, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(1L);
+ }
+
+ addProbeRecord(h, 3L, "k3", "no-match");
+ assertThat(op.getMaxJoinFanOutGauge().getValue()).isEqualTo(3L);
+ if (leftOuter) {
+ // fan-out 1 (LEFT OUTER, null-padded)
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(6.0d / 4, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(0L);
+ } else {
+ // fan-out 0 (INNER, unmatched)
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(4.0d / 4, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(2L);
+ }
+ }
+ }
+
+ @Test
+ void numUnmatchedBuildRetractionsCountsAbsentRowRetractions() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // LOAD: a -D for a never-inserted row is counted.
+ addBuildChange(h, deleteRecord("k1", "ghost", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 2L));
+ assertThat(op.getNumUnmatchedBuildRetractions().getCount()).isEqualTo(0L);
Review Comment:
Stepping back on coverage - the two correctness properties the operator is really built around could use direct tests:
- **ref-count underflow**: right now only the metric is asserted - could we also check the multiset map itself, e.g. that a `-D` on a count-1 row removes the key rather than going negative?
- **atomic `-U`/`+U` visibility mid-buffer**: it'd be great to have a case where a probe joins *while* a `-U`/`+U` pair is still buffered (should see the old value), then re-probes after the watermark advances (sees the new value) - that's really the whole point of the buffer, so worth pinning down
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperatorTest.java:
##########
@@ -0,0 +1,1666 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.KeyedTwoInputStreamOperatorTestHarness;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.data.writer.BinaryRowWriter;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.Phase;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.runtime.util.RowDataHarnessAssertor;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.flink.types.RowKind;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_CHANGE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_TABLE_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.PROBE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.deleteRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.insertRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateAfterRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateBeforeRecord;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.within;
+
+/** Harness tests for {@link LateralSnapshotJoinOperator}. */
+class LateralSnapshotJoinOperatorTest {
+
+ // ----------------------------------------------------------------- Schema
+
+ /** Probe row schema: (id BIGINT, key VARCHAR, val VARCHAR). */
+ private static final InternalTypeInfo<RowData> PROBE_TYPE =
+ InternalTypeInfo.ofFields(
+ new BigIntType(), VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+
+ /** Build row schema: (key VARCHAR, val VARCHAR, rt BIGINT). */
+ private static final InternalTypeInfo<RowData> BUILD_TYPE =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE, VarCharType.STRING_TYPE, new BigIntType());
+
+ /** Joined output schema: probe ++ build = (id, pKey, pVal, bKey, bVal, bRt). */
+ private static final LogicalType[] OUTPUT_TYPES = {
+ new BigIntType(),
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType()
+ };
+
+ /** Probe key column index (key VARCHAR is at field 1). */
+ private static final int PROBE_KEY_IDX = 1;
+
+ /** Build key column index (key VARCHAR is at field 0). */
+ private static final int BUILD_KEY_IDX = 0;
+
+ /** Build row-time column index (rt BIGINT is at field 2). */
+ private static final int BUILD_RT_IDX = 2;
+
+ private static final InternalTypeInfo<RowData> KEY_TYPE =
+ InternalTypeInfo.ofFields(VarCharType.STRING_TYPE);
+
+ private static final KeySelector<RowData, RowData> PROBE_KEY_SELECTOR =
+ nullSafeStringKeySelector(PROBE_KEY_IDX);
+ private static final KeySelector<RowData, RowData> BUILD_KEY_SELECTOR =
+ nullSafeStringKeySelector(BUILD_KEY_IDX);
+
+ private static final RowDataHarnessAssertor JOINED_ASSERTOR =
+ new RowDataHarnessAssertor(OUTPUT_TYPES);
+
+ // ----------------------------------------------------------------- Join conditions
+
+ /** Trivial join condition that always matches (equality is enforced by partitioning). */
+ private static final String ALWAYS_TRUE_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionStub extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionStub(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) { return true; }\n"
+ + "}\n";
+
+ /**
+ * Join condition that only matches when the probe value (field 2) equals {@code "match"}. Used
+ * to verify that the codegen'd condition is actually invoked at join time.
+ */
+ private static final String MATCH_VAL_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionMatchVal extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionMatchVal(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) {\n"
+ + " if (in1.isNullAt(2)) { return false; }\n"
+ + " return \"match\".equals(in1.getString(2).toString());\n"
+ + " }\n"
+ + "}\n";
+
+ private static GeneratedJoinCondition newTrueCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionStub", ALWAYS_TRUE_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ private static GeneratedJoinCondition newMatchValCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionMatchVal", MATCH_VAL_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ // ----------------------------------------------------------------- Operator / harness
+ // factories
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ GeneratedJoinCondition joinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return new LateralSnapshotJoinOperator(
+ isLeftOuterJoin,
+ PROBE_TYPE,
+ BUILD_TYPE,
+ BUILD_RT_IDX,
+ joinCondition,
+ filterNullKeys,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return newOperator(
+ isLeftOuterJoin,
+ newTrueCondition(),
+ new boolean[] {true},
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData>
+ newHarness(LateralSnapshotJoinOperator op) throws Exception {
+ return new KeyedTwoInputStreamOperatorTestHarness<>(
+ op, PROBE_KEY_SELECTOR, BUILD_KEY_SELECTOR, KEY_TYPE);
+ }
+
+ private static KeySelector<RowData, RowData> nullSafeStringKeySelector(final int keyIdx) {
+ return value -> {
+ BinaryRowData ret = new BinaryRowData(1);
+ BinaryRowWriter writer = new BinaryRowWriter(ret);
+ if (value.isNullAt(keyIdx)) {
+ writer.setNullAt(0);
+ } else {
+ writer.writeString(0, value.getString(keyIdx));
+ }
+ writer.complete();
+ return ret;
+ };
+ }
+
+ // ----------------------------------------------------------------- LOAD phase
+
+ @Test
+ void loadPhaseBuildSideChangeProcessing() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // During LOAD, build-side changes are buffered and later applied in event-time order.
+ // -D for a never-inserted (key, value) pair is defensively ignored.
+ addBuildChange(h, deleteRecord("k1", "ghost", 5L));
+ // Two identical records (same key/val/row-time) → count(k1, v1, 20) = 2.
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ // Earlier row-time than v1, but arrives later.
+ addBuildChange(h, insertRecord("k1", "v2", 10L));
+
+ // Still LOAD: changes are buffered, nothing applied or emitted yet.
+ assertPhase(op, Phase.LOAD);
+ assertThat(h.getOutput()).isEmpty();
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(buildTableKeys(h)).isEmpty();
+
+ // Advance the build watermark (still below the flip point) and access k1 again. The
+ // access drains the buffered batch in event-time order before buffering the new change.
+ addBuildWm(h, 50L);
+ addBuildChange(h, insertRecord("k1", "v3", 30L));
+ // TODO: also add updateBefore and updateAfter changes, to ensure that these are handled
+ // correctly
+
+ assertPhase(op, Phase.LOAD);
+ // Applied in row-time order: -D(ghost)@5 (ignored), +I(v2)@10, +I(v1)@20 ×2.
+ assertThat(buildTableKeys(h)).containsExactly("k1");
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L));
+ // The triggering v3 change is now buffered, awaiting the next drain.
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(1);
+ }
+ }
+
+ @Test
+ void loadPhaseProbeSideInputProcessing() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+
+ addBuildChange(h, insertRecord("k1", "build1", 1L));
+ addBuildWm(h, 20L);
+ addProbeRecord(h, 1L, "k1", "probe-load-1");
+ addProbeRecord(h, 2L, "k1", "probe-load-2");
+ addProbeWm(h, 50L);
+
+ assertPhase(op, Phase.LOAD);
+ // No output (records buffered, watermarks held back).
+ assertThat(h.getOutput()).isEmpty();
+
+ assertThat(op.getCurrentProbeSideWm()).isEqualTo(50L);
+ assertThat(probeBufferKeys(h)).containsExactly("k1");
+ assertThat(probeBufferForKey(h, op, "k1")).hasSize(2);
+ // Build-side change was applied to the build table.
+ assertThat(bufferedChangesForKey(h, op, "k1")).isEmpty();
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("build1", 1L));
+ }
+ }
+
+ // ----------------------------------------------------------------- Flip / transition
+
+ @ParameterizedTest(name = "leftOuter={0}, wmFlip={1}")
+ @CsvSource({"true, true", "true, false", "false, true", "false, false"})
+ void flipDrainsProbeBufferAndJoins(boolean leftOuter, boolean wmFlip) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(leftOuter, 100L, 200L, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ // LOAD: build state and buffered probes (one without a matching build row).
+ // k1 is a multi-set: count(v1)=2, count(v2)=1; k2 is a single row.
+ addBuildChange(h, insertRecord("k1", "build-k1-v1", 1L));
+ addBuildChange(h, insertRecord("k1", "build-k1-v1", 1L));
+ addBuildChange(h, insertRecord("k1", "build-k1-v2", 1L));
+ addBuildChange(h, insertRecord("k2", "build-k2", 1L));
+ // LOAD: add probe-side records
+ addProbeRecord(h, 1L, "k1", "probe-1");
+ addProbeRecord(h, 2L, "k2", "probe-2");
+ addProbeRecord(h, 3L, "k2", "probe-3");
+ addProbeRecord(h, 4L, "k3", "probe-no-match");
+ addProbeWm(h, 80L);
+ assertPhase(op, Phase.LOAD);
+
+ // assert that probe-side buffer is filled
+ assertThat(probeBufferForKey(h, op, "k1")).hasSize(1);
+ assertThat(probeBufferForKey(h, op, "k2")).hasSize(2);
+ assertThat(probeBufferForKey(h, op, "k3")).hasSize(1);
+ // idle-timeout timer is armed while in LOAD
+ assertThat(op.isIdleFlipTimerActive()).isTrue();
+
+ // trigger flip from LOAD to JOIN
+ if (wmFlip) {
+ // build WM crosses loadCompletedTime.
+ addBuildWm(h, 100L);
+ } else {
+ // proc-time exceeds idle timeout
+ h.setProcessingTime(200);
+ }
+ assertPhase(op, Phase.JOIN);
+ // idle-timeout timer is removed on flip (canceled by a WM flip, fired by an idle flip)
+ assertThat(op.isIdleFlipTimerActive()).isFalse();
+
+ // probe k1 joins k1's multi-set (count-respecting: 2x v1 + 1x v2 = three rows);
+ // probe k2 (2 rows) joins a single k2 build row;
+ // probe k3 doesn't have a matching build row. INNER: no output, LEFT OUTER: null-padded
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 80L);
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v2", 1L),
+ row(2L, "k2", "probe-2", "k2", "build-k2", 1L),
+ row(3L, "k2", "probe-3", "k2", "build-k2", 1L),
+ row(4L, "k3", "probe-no-match", null, null, null));
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v2", 1L),
+ row(2L, "k2", "probe-2", "k2", "build-k2", 1L),
+ row(3L, "k2", "probe-3", "k2", "build-k2", 1L));
+ }
+ // Probe buffer drained on flip; build table preserved (k1 keeps multi-set counts).
+ assertThat(probeBufferKeys(h)).isEmpty();
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(
+ Map.of("build-k1-v1", 2L, "build-k1-v2", 1L));
+ assertThat(buildTableForKey(h, op, "k2"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("build-k2", 1L));
+ }
+ }
+
+ @Test
+ void idleTimerRearmsOnBuildWatermark() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 1000L, 100L, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(10);
+ h.open();
+ h.setProcessingTime(60);
+ // Build WM advances → re-arm.
+ addBuildWm(h, 10L);
+ // Original idle deadline was 10+100=110. Re-armed to 60+100=160.
+ h.setProcessingTime(159);
+ assertPhase(op, Phase.LOAD);
+ h.setProcessingTime(160);
+ assertPhase(op, Phase.JOIN);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void flipJoiningInvokesCodeGeneratedJoinCondition(boolean leftOuter) throws Exception {
+ LateralSnapshotJoinOperator op =
+ newOperator(
+ leftOuter, newMatchValCondition(), new boolean[] {true}, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addProbeRecord(h, 1L, "k1", "match");
+ addProbeRecord(h, 2L, "k1", "skip");
+ addProbeWm(h, 120L);
+ addBuildWm(h, 100L);
+
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 120L);
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "match", "k1", "v1", 1L),
+ row(2L, "k1", "skip", null, null, null));
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, "k1", "match", "k1", "v1", 1L));
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void flipJoiningCompositeEquiKeys(boolean leftOuter) throws Exception {
+ // Probe schema (kA VARCHAR, kB VARCHAR, val VARCHAR); build schema additionally carries a
+ // row-time attribute (kA VARCHAR, kB VARCHAR, val VARCHAR, rt BIGINT) at index 3.
+ InternalTypeInfo<RowData> probeType =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE, VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+ InternalTypeInfo<RowData> buildType =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType());
+ final int buildRowtimeIndex = 3;
+ // Compose the composite key as a BinaryRowData with both key fields.
+ KeySelector<RowData, RowData> selector =
+ value -> {
+ BinaryRowData ret = new BinaryRowData(2);
+ BinaryRowWriter writer = new BinaryRowWriter(ret);
+ if (value.isNullAt(0)) {
+ writer.setNullAt(0);
+ } else {
+ writer.writeString(0, value.getString(0));
+ }
+ if (value.isNullAt(1)) {
+ writer.setNullAt(1);
+ } else {
+ writer.writeString(1, value.getString(1));
+ }
+ writer.complete();
+ return ret;
+ };
+ InternalTypeInfo<RowData> compositeKeyType =
+ InternalTypeInfo.ofFields(VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+
+ LateralSnapshotJoinOperator op =
+ new LateralSnapshotJoinOperator(
+ leftOuter,
+ probeType,
+ buildType,
+ buildRowtimeIndex,
+ newTrueCondition(),
+ new boolean[] {true, true},
+ 100L,
+ null,
+ null);
+
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ new KeyedTwoInputStreamOperatorTestHarness<>(
+ op, selector, selector, compositeKeyType)) {
+ h.open();
+ // build rows: two identical rows for key a-1, a different row for key a-1, one row for
+ // key a-2.
+ addBuildChange(h, insertRecord("a", "1", "b-a-1-1", 1L));
+ addBuildChange(h, insertRecord("a", "1", "b-a-1-1", 1L));
+ addBuildChange(h, insertRecord("a", "1", "b-a-1-2", 1L));
+ addBuildChange(h, insertRecord("a", "2", "b-a-2-1", 1L));
+ // probes: matching composite, non-matching composite.
+ h.processElement1(insertRecord("a", "1", "p-a-1"));
+ h.processElement1(insertRecord("a", "9", "p-a-9"));
+ h.processElement1(insertRecord("b", "1", "p-b-1"));
+ h.processElement1(insertRecord("b", "9", "p-b-9"));
+ // flip to JOIN phase
+ addBuildWm(h, 100L);
+
+ LogicalType[] outTypes = {
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType()
+ };
+ RowDataHarnessAssertor compositeAssertor = new RowDataHarnessAssertor(outTypes);
+
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ compositeAssertor.shouldEmitAll(
+ h,
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-2", 1L),
+ compKeyRow("a", "9", "p-a-9", null, null, null, null),
+ compKeyRow("b", "1", "p-b-1", null, null, null, null),
+ compKeyRow("b", "9", "p-b-9", null, null, null, null));
+ } else {
+ compositeAssertor.shouldEmitAll(
+ h,
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-2", 1L));
+ }
+ }
+ }
+
+ // ----------------------------------------------------------------- JOIN phase
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void joinPhaseImmediateInnerJoin(boolean leftOuter) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(leftOuter, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // First probe WM
+ addProbeWm(h, 10L);
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 10L);
+
+ // First probe — joined immediately.
+ addProbeRecord(h, 1L, "k1", "probe-immediate-1");
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, "k1", "probe-immediate-1", "k1", "v1", 1L));
+
+ // Another probe WM
+ addProbeWm(h, 20L);
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 20L);
+
+ // Second probe for same key — joined immediately.
+ addProbeRecord(h, 2L, "k1", "probe-immediate-2");
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(h, row(2L, "k1", "probe-immediate-2", "k1", "v1", 1L));
+
+ // Probe for non-existent key — no output (INNER).
+ addProbeRecord(h, 3L, "k2", "probe-no-match");
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(3L, "k2", "probe-no-match", null, null, null));
+ } else {
+ assertThat(h.extractOutputStreamRecords()).isEmpty();
+ }
+
+ // one more probe WM
+ addProbeWm(h, 30L);
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 30L);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void joinPhaseBuildSideChangeApplication(boolean appliedByBuild) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // Two identical records for k1 (count 2) at row-time 1, buffered during LOAD.
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // Buffer four changes for k1
+ addBuildChange(h, deleteRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v2", 102L));
+ addBuildChange(h, updateBeforeRecord("k1", "v1", 1L));
+ addBuildChange(h, updateAfterRecord("k1", "v3", 103L));
+ // Buffer one change for k2.
+ addBuildChange(h, insertRecord("k2", "v1", 101L));
+
+ // assert number of buffered changes
+ assertThat(op.getCurrentBuildSideWm()).isEqualTo(100L);
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(bufferedAtWmFor(h, op, "k2")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k2")).hasSize(1);
+ assertThat(buildTableForKey(h, op, "k1")).isEqualTo(Map.of("v1", 2L));
+
+ // Probe record with no build WM advance - changes are not applied yet
+ addProbeRecord(h, 1L, "k1", "p-1");
+ JOINED_ASSERTOR.shouldEmitAll(
+ h, row(1L, "k1", "p-1", "k1", "v1", 1L), row(1L, "k1", "p-1", "k1", "v1", 1L));
+
+ // increment build-side WM
+ addBuildWm(h, 110L);
+
+ // assert that all changes are still buffered
+ assertThat(op.getCurrentBuildSideWm()).isEqualTo(110L);
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(bufferedAtWmFor(h, op, "k2")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k2")).hasSize(1);
+
+ // trigger application of k1 changes by build or probe-side input
+ if (!appliedByBuild) {
+ addBuildChange(h, insertRecord("k1", "v4", 111L));
+ // assert that changes have been applied and removed from buffer
+ // the triggering change is appended to the buffer
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(110L);
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(1);
+ assertThat(buildTableForKey(h, op, "k1")).isEqualTo(Map.of("v2", 1L, "v3", 1L));
+ // assert empty output
+ assertThat(h.getOutput()).isEmpty();
+ } else {
+ addProbeRecord(h, 2L, "k1", "p-2");
+ // assert that changes have been applied and removed from buffer
+ assertThat(bufferedAtWmFor(h, op, "k1")).isNull();
+ assertThat(bufferedChangesForKey(h, op, "k1")).isEmpty();
+ assertThat(buildTableForKey(h, op, "k1"))
+ .isEqualTo(Map.of("v2", 1L, "v3", 1L)); // /
+ // assert join results (v2 carries row-time 2, v3 carries row-time 1)
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(2L, "k1", "p-2", "k1", "v2", 102L),
+ row(2L, "k1", "p-2", "k1", "v3", 103L));
+ }
+
+ // assert that k2 change is still buffered
+ assertThat(bufferedAtWmFor(h, op, "k2")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k2")).hasSize(1);
+ // apply k2 change and join
+ addProbeRecord(h, 3L, "k2", "p-3");
+ assertThat(bufferedAtWmFor(h, op, "k2")).isNull();
+ assertThat(bufferedChangesForKey(h, op, "k2")).isEmpty();
+ assertThat(buildTableForKey(h, op, "k2")).isEqualTo(Map.of("v1", 1L));
+ // assert join results
+ JOINED_ASSERTOR.shouldEmitAll(h, row(3L, "k2", "p-3", "k2", "v1", 101L));
+ }
+ }
+
+ @Test
+ void joinPhaseWmForwardingLogic() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L); // flip
+ assertPhase(op, Phase.JOIN);
+
+ // Build-side WMs after flip are not forwarded.
+ addBuildWm(h, 200L);
+ assertThat(extractWatermarks(h.getOutput())).isEmpty();
+
+ // Probe-side WMs in JOIN are forwarded.
+ addProbeWm(h, 150L);
+ assertThat(extractWatermarks(h.getOutput())).containsExactly(new Watermark(150));
+ h.getOutput().clear();
+
+ // another build-side WM
+ addBuildWm(h, 300L);
+ assertThat(extractWatermarks(h.getOutput())).isEmpty();
+
+ addProbeWm(h, 250L);
+ assertThat(extractWatermarks(h.getOutput())).containsExactly(new Watermark(250));
+ }
+ }
+
+ /**
+ * For buffered build-side changes sharing the same row-time, retractions are applied before
+ * accumulations.
+ */
+ @Test
+ void joinPhaseAppliesRetractionsBeforeAccumulationsAtEqualRowTime() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // Drive into JOIN with empty build state for k1/k2.
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // Buffer accumulation-first at row-time 105 on absent rows:
+ // k1: +I then -D ; k2: +U then -U ; k3: +U then -U
+ addBuildChange(h, insertRecord("k1", "v1", 105L));
+ addBuildChange(h, deleteRecord("k1", "v1", 105L));
+ addBuildChange(h, updateAfterRecord("k2", "v1", 105L));
+ addBuildChange(h, updateBeforeRecord("k2", "v1", 105L));
+
+ // Advance build WM, then access each key to drain its buffer in event-time order.
+ addBuildWm(h, 200L);
+ addProbeRecord(h, 1L, "k1", "p1");
+ addProbeRecord(h, 2L, "k2", "p2");
+
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+ assertThat(buildTableForKey(h, op, "k2"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 105L),
+ row(2L, "k2", "p2", "k2", "v1", 105L));
+ }
+ }
+
+ // ----------------------------------------------------------------- NULL keys
+
+ @ParameterizedTest
+ @CsvSource({"true, true", "true, false", "false, true", "false, false"})
+ void joinRespectsNullKeysFilter(boolean leftOuter, boolean filterNullKey) throws Exception {
+ LateralSnapshotJoinOperator op =
+ newOperator(
+ leftOuter,
+ newTrueCondition(),
+ new boolean[] {filterNullKey},
+ 100L,
+ null,
+ null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord(null, "v_null", 1L));
+ addProbeRecord(h, 1L, null, "p_null");
+ addBuildWm(h, 100L);
+
+ // test joining during LOAD -> JOIN transition
+ if (filterNullKey) {
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, null, "p_null", null, null, null));
+ } else {
+ assertThat(h.getOutput()).isEmpty();
+ }
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, null, "p_null", null, "v_null", 1L));
+ }
+
+ // test joining in JOIN phase
+ addProbeRecord(h, 2L, null, "p_null");
+ if (filterNullKey) {
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(2L, null, "p_null", null, null, null));
+ } else {
+ assertThat(h.getOutput()).isEmpty();
+ }
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(2L, null, "p_null", null, "v_null", 1L));
+ }
+ }
+ }
+
+ // ----------------------------------------------------------------- Watermark status
+
+ @Test
+ void buildSideWmAndWmStatusForwarding() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // probe-side WM
+ addProbeWm(h, 70L);
+
+ // LOAD phase: build-side becomes idle then active again. WM is advanced.
+ h.processWatermarkStatus2(WatermarkStatus.IDLE);
+ h.processWatermarkStatus2(WatermarkStatus.ACTIVE);
+ addBuildWm(h, 50L);
+ // assert that no WMs or WM statuses are emitted in LOAD.
+ assertThat(h.getOutput()).isEmpty();
+
+ // Drive into JOIN
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // assert that only the probe-side WM was forwarded after the transition
+ assertThat(h.getOutput()).containsExactly(new Watermark(70L));
+ h.getOutput().clear();
+
+ // JOIN phase: build-side becomes idle then active again. WM is advanced
+ h.processWatermarkStatus2(WatermarkStatus.IDLE);
+ h.processWatermarkStatus2(WatermarkStatus.ACTIVE);
+ addBuildWm(h, 200L);
+
+ // No records, WMs, or WM statuses emitted in JOIN.
+ assertThat(h.getOutput()).isEmpty();
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void probeSideWmAndWmStatusForwarding(boolean probeIdleDuringLoad) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addProbeWm(h, 25L);
+ addProbeWm(h, 50L);
+ // Probe-side WM statuses received during LOAD.
+ h.processWatermarkStatus1(WatermarkStatus.IDLE);
+ if (!probeIdleDuringLoad) {
+ h.processWatermarkStatus1(WatermarkStatus.ACTIVE);
+ }
+ // Absorbed during LOAD — nothing emitted.
+ assertThat(extractWatermarks(h.getOutput())).isEmpty();
+ assertThat(extractWatermarkStatuses(h.getOutput())).isEmpty();
+
+ // Flip to JOIN phase — last probe WM emitted.
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ if (probeIdleDuringLoad) {
+ assertThat(extractWatermarkStatuses(h.getOutput()))
+ .containsExactly(WatermarkStatus.IDLE);
+ } else {
+ assertThat(extractWatermarkStatuses(h.getOutput())).isEmpty();
+ }
+ assertThat(extractWatermarks(h.getOutput())).containsExactly(new Watermark(50));
+ h.getOutput().clear();
+
+ // WMs and WM status updates received during JOIN are forwarded.
+ addProbeWm(h, 150L);
+ assertThat(h.getOutput()).containsExactly(new Watermark(150));
+ h.getOutput().clear();
+ // set probe-side to idle
+ h.processWatermarkStatus1(WatermarkStatus.IDLE);
+ assertThat(h.getOutput()).containsExactly(WatermarkStatus.IDLE);
+ h.getOutput().clear();
+ // set probe-side to active
+ h.processWatermarkStatus1(WatermarkStatus.ACTIVE);
+ assertThat(h.getOutput()).containsExactly(WatermarkStatus.ACTIVE);
+ h.getOutput().clear();
+ // emit another watermark
+ addProbeWm(h, 200L);
+ assertThat(h.getOutput()).containsExactly(new Watermark(200));
+ h.getOutput().clear();
+ }
+ }
+
+ // ----------------------------------------------------------------- State TTL
+
+ @Test
+ void stateTtlRefreshesOnAccessAndEvictsInactiveKeys() throws Exception {
+ // stateTtlMs = 100. Timers are registered at 1.5 × stateTtlMs, so the deadline for access
+ // at t=0 is 150.
+ LateralSnapshotJoinOperator op = newOperator(false, 50L, null, 100L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ addBuildChange(h, insertRecord("k3", "v1", 1L));
+ addBuildChange(h, insertRecord("k4", "v1", 1L));
+
+ // State is NOT evicted during LOAD even after the deadline passes (TTL fires are
+ // rescheduled past the LOAD phase).
+ h.setProcessingTime(200);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3", "k4");
+
+ // flip to JOIN
+ addBuildWm(h, 50L);
+ assertPhase(op, Phase.JOIN);
+
+ // Touch k1, k2, k3 to reset their TTL in JOIN; leave k4 alone.
+ h.setProcessingTime(275);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3", "k4");
+ addBuildChange(h, insertRecord("k1", "v2", 2L));
+ addBuildChange(h, insertRecord("k2", "v2", 2L));
+ addBuildChange(h, insertRecord("k3", "v2", 2L));
+
+ // k4 evicted: it wasn't accessed since proc-time (0) and we flipped to JOIN at (200)
+ h.setProcessingTime(350);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3");
+
+ // Update build state for k1 and k2; leave k3 alone
+ addBuildWm(h, 60L);
+ h.setProcessingTime(400);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3");
+ addBuildChange(h, insertRecord("k1", "v3", 3L));
+ addBuildChange(h, insertRecord("k2", "v3", 3L));
+
+ // k3 evicted, not accessed since (275)
+ addBuildWm(h, 70L);
+ h.setProcessingTime(475);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2");
+ // Access k1 from probe side to reset its TTL again; leave k2 alone
+ addProbeRecord(h, 1L, "k1", "p1");
+
+ // k2 evicted, not accessed since (400)
+ h.setProcessingTime(550);
+ assertThat(buildStateKeys(h)).containsExactly("k1");
+
+ // k1 finally evicted.
+ h.setProcessingTime(700);
+ assertThat(buildStateKeys(h)).isEmpty();
+
+ // The probe joined against the (k1) multi-set state at proc-time 475 — entries v1,
+ // v2, v3.
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v2", 2L),
+ row(1L, "k1", "p1", "k1", "v3", 3L));
+ }
+ }
+
+ @Test
+ void stateTtlClearsAllPerKeyState() throws Exception {
+ // stateTtlMs = 100. Timers are registered at 1.5 × stateTtlMs.
+ LateralSnapshotJoinOperator op = newOperator(false, 50L, null, 100L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ // Buffered during LOAD
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 50L); // flip to JOIN
+ assertPhase(op, Phase.JOIN);
+
+ // Buffer a change in JOIN that does NOT drain (no further WM advance / access). This
+ // populates the change buffer and the buffered-at tag while the build table keeps {v1}.
+ addBuildChange(h, insertRecord("k1", "v2"));
+ assertThat(buildTableForKey(h, op, "k1")).isEqualTo(Map.of("v1", 1L));
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(1);
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(50L);
+ assertThat(ttlExpiryFor(h, op, "k1")).isEqualTo(150L);
+
+ // Fire the TTL timer (well past the eviction deadline).
+ h.setProcessingTime(1000);
+
+ // Every per-key state object is cleared, not just the build table.
+ assertThat(buildTableForKey(h, op, "k1")).isEmpty();
+ assertThat(bufferedChangesForKey(h, op, "k1")).isEmpty();
+ assertThat(bufferedAtWmFor(h, op, "k1")).isNull();
+ assertThat(probeBufferForKey(h, op, "k1")).isEmpty();
+ assertThat(ttlExpiryFor(h, op, "k1")).isNull();
+ }
+ }
+
+ @Test
+ void stateTtlRestoreResetsFlipProcTime() throws Exception {
+ // stateTtlMs = 50. Build write at t=0 arms TTL at 75. Flip at t=0 → flipProcTime=0.
+ // Original grace ends at 0+50=50.
+ LateralSnapshotJoinOperator op1 = newOperator(false, 100L, null, 50L);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.setProcessingTime(0);
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertThat(op1.getFlipProcTime()).isEqualTo(0L);
+ state = h.snapshot(0L, 0L);
+ }
+
+ // Restart at t=30 — flipProcTime is re-anchored to 30; new grace ends at 30+50=80.
+ LateralSnapshotJoinOperator op2 = newOperator(false, 100L, null, 50L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.setProcessingTime(30);
+ h.initializeState(state);
+ h.open();
+ assertThat(op2.getFlipProcTime()).isEqualTo(30L);
+
+ // At t=75 the recovered TTL timer fires. Grace check: now=75 < flipProcTime+stateTtlMs
+ // = 80 → reschedule rather than evict.
+ h.setProcessingTime(75);
+
+ // k1 still present because the grace window was re-anchored.
+ assertThat(buildStateKeys(h)).containsExactly("k1");
+
+ // advance proc-time to 105 to evict state
+ h.setProcessingTime(105);
+ assertThat(buildStateKeys(h)).isEmpty();
+ }
+ }
+
+ // ----------------------------------------------------------------- Snapshot / restore
+
+ @Test
+ void restoreFromLoadPhaseSnapshot() throws Exception {
+ LateralSnapshotJoinOperator op1 = newOperator(false, 100L, null, null);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.open();
+ // Build-side multi-set with a duplicate count, plus a buffered
+ // probe.
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v2", 2L));
+ addBuildWm(h, 10L);
+ addBuildChange(h, insertRecord("k1", "v3", 3L));
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertPhase(op1, Phase.LOAD);
+ assertThat(probeBufferKeys(h)).containsExactly("k1");
+ state = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator op2 = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.initializeState(state);
+ h.open();
+ assertPhase(op2, Phase.LOAD);
+ // Buffered probe and build-table multi-set (with counts) preserved across restore.
+ assertThat(probeBufferKeys(h)).containsExactly("k1");
+ assertThat(buildTableForKey(h, op2, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L));
+ assertThat(bufferedChangesForKey(h, op2, "k1")).hasSize(1);
+ // Gauge is rebuilt from restored keyed state: one probe buffered for k1.
+ assertThat(op2.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(1L);
+
+ // Trigger flip; the buffered probe is joined post-restore, count-respecting against the
+ // restored multi-set (2× v1 + 1× v2 = three rows).
+ addProbeWm(h, 50L);
+ addBuildWm(h, 100L);
+ assertPhase(op2, Phase.JOIN);
+ assertThat(probeBufferKeys(h)).isEmpty();
+ // Draining the restored probe and build buffers brings both gauges back to 0.
+ assertThat(op2.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ assertThat(buildTableForKey(h, op2, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L, "v3", 1L));
+
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v2", 2L),
+ row(1L, "k1", "p1", "k1", "v3", 3L));
+ }
+ }
+
+ @Test
+ void restoreFromMixedPhaseSnapshot() throws Exception {
+ // Subtask A: drive into JOIN with a buffered change for k1.
+ LateralSnapshotJoinOperator opA = newOperator(false, 100L, null, null);
+ OperatorSubtaskState stateA;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(opA)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(opA, Phase.JOIN);
+ // Buffer a build-side change
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ stateA = h.snapshot(0L, 0L);
+ }
+
+ // Subtask B: stay in LOAD (no flip-triggering build WM).
+ LateralSnapshotJoinOperator opB = newOperator(false, 100L, null, null);
+ OperatorSubtaskState stateB;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(opB)) {
+ h.open();
+ addProbeRecord(h, 1L, "k2", "p1");
+ assertPhase(opB, Phase.LOAD);
+ stateB = h.snapshot(0L, 0L);
+ }
+
+ OperatorSubtaskState combined =
+ AbstractStreamOperatorTestHarness.repackageState(stateA, stateB);
+
+ // Restore the combined state — phase must be LOAD because some subtask was LOAD.
+ LateralSnapshotJoinOperator opC = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(opC)) {
+ h.initializeState(combined);
+ h.open();
+ assertPhase(opC, Phase.LOAD);
+
+ // assert state: A's build table ({v1:1}) plus A's buffered change and B's probe.
+ assertThat(buildTableKeys(h)).containsExactly("k1");
+ assertThat(buildTableForKey(h, opC, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+ assertThat(bufferedChangesForKey(h, opC, "k1")).hasSize(1);
+ assertThat(probeBufferKeys(h)).containsExactly("k2");
+
+ // During LOAD the current build WM is still MIN_VALUE, so the recovered buffer (tagged
+ // at the pre-restore WM) is not drained; the new change is appended to it.
+ addBuildChange(h, insertRecord("k1", "v2", 2L));
+ assertThat(bufferedChangesForKey(h, opC, "k1")).hasSize(2);
+ assertThat(buildTableForKey(h, opC, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+
+ // Trigger flip
+ addBuildWm(h, 100L);
+ assertPhase(opC, Phase.JOIN);
+
+ // Access k1: the buffered changes drain in event-time order, then the probe joins.
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(bufferedChangesForKey(h, opC, "k1")).isEmpty();
+ assertThat(buildTableForKey(h, opC, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L));
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v2", 2L));
+ }
+ }
+
+ @Test
+ void restoreFromJoinPhaseSnapshot() throws Exception {
+ LateralSnapshotJoinOperator op1 = newOperator(false, 100L, null, null);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op1, Phase.JOIN);
+ // Buffer a -U/+U pair for each key (tagged at bufferedAt = 100).
+ addBuildChange(h, updateBeforeRecord("k1", "v1", 1L));
+ addBuildChange(h, updateAfterRecord("k1", "v2", 101L));
+ addBuildChange(h, updateBeforeRecord("k2", "v1", 1L));
+ addBuildChange(h, updateAfterRecord("k2", "v2", 101L));
+ state = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator op2 = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.initializeState(state);
+ h.open();
+ assertPhase(op2, Phase.JOIN);
+ assertThat(op2.getCurrentBuildSideWm()).isEqualTo(Long.MIN_VALUE);
+ assertThat(bufferedChangesForKey(h, op2, "k1")).hasSize(2);
+ assertThat(bufferedChangesForKey(h, op2, "k2")).hasSize(2);
+ // Gauge is rebuilt from restored keyed state: four build-side changes buffered
+ // (two for k1, two for k2).
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(4L);
+
+ // k2: accessed while no build WM has arrived since restore → eager drain.
+ addProbeRecord(h, 1L, "k2", "p-k2");
+ assertThat(bufferedChangesForKey(h, op2, "k2")).isEmpty();
+ // Draining k2's two restored changes leaves k1's two still buffered.
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(2L);
+ assertThat(buildTableForKey(h, op2, "k2"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v2", 1L));
+
+ // k1: still buffered (eager drain of k2 left latestBuildSideWm at MIN_VALUE); advance
+ // the build WM, then access → normal WM-gated drain.
+ assertThat(bufferedChangesForKey(h, op2, "k1")).hasSize(2);
+ addBuildWm(h, 200L);
+ addProbeRecord(h, 2L, "k1", "p-k1");
+ assertThat(bufferedChangesForKey(h, op2, "k1")).isEmpty();
+ // All restored changes drained; gauge back to 0.
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ assertThat(buildTableForKey(h, op2, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v2", 1L));
+
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k2", "p-k2", "k2", "v2", 101L),
+ row(2L, "k1", "p-k1", "k1", "v2", 101L));
+ }
+ }
+
+ @Test
+ void restoreRearmsIdleFlipTimer() throws Exception {
+ LateralSnapshotJoinOperator op1 = newOperator(false, 1000L, 100L, null);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.setProcessingTime(50);
+ h.open();
+ assertPhase(op1, Phase.LOAD);
+ state = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator op2 = newOperator(false, 1000L, 100L, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.setProcessingTime(150);
+ h.initializeState(state);
+ h.open();
+ assertPhase(op2, Phase.LOAD);
+ // Re-armed at open()'s proc-time + idleTimeout = 150 + 100 = 250.
+ h.setProcessingTime(249);
+ assertPhase(op2, Phase.LOAD);
+ h.setProcessingTime(250);
+ assertPhase(op2, Phase.JOIN);
+ }
+ }
+
+ // ----------------------------------------------------------------- Metrics
+
+ @Test
+ void currentPhaseGaugeReflectsPhase() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // LOAD = ordinal 0.
+ assertThat(op.getCurrentPhaseGauge().getValue()).isEqualTo(0);
+
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+ // JOIN = ordinal 1.
+ assertThat(op.getCurrentPhaseGauge().getValue()).isEqualTo(1);
+ }
+ }
+
+ @Test
+ void probeBufferedGaugeTracksLoadBufferingAndDrainsOnFlip() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(1L);
+ addProbeRecord(h, 2L, "k2", "p2");
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(2L);
+ addProbeRecord(h, 3L, "k1", "p3");
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+
+ // Flip drains all buffered probes across keys.
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ }
+ }
+
+ @Test
+ void buildBufferedGaugeTracksJoinBufferingAndDrains() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // Flip to JOIN with an empty build state (no build records during LOAD).
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+
+ // JOIN-phase build changes are buffered (their tag equals the current build WM).
+ addBuildChange(h, insertRecord("k1", "v2", 101L));
+ addBuildChange(h, insertRecord("k1", "v3", 101L));
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(2L);
+ addBuildChange(h, insertRecord("k2", "v1", 102L));
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+
+ // Advance build WM, then drain each key by accessing it from the probe side.
+ addBuildWm(h, 110L);
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(1L);
+ addProbeRecord(h, 2L, "k2", "p2");
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ }
+ }
+
+ @Test
+ void watermarkGaugesTrackBuildAndProbeWatermarks() throws Exception {
+ // High loadCompletedTime so the operator stays in LOAD for the first build WMs.
+ LateralSnapshotJoinOperator op = newOperator(false, 1000L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(Long.MIN_VALUE);
+ assertThat(op.getCurrentProbeSideWatermarkGauge().getValue()).isEqualTo(Long.MIN_VALUE);
+
+ // LOAD phase: both watermarks are tracked even though nothing is forwarded.
+ addBuildWm(h, 50L);
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(50L);
+ addProbeWm(h, 70L);
+ assertThat(op.getCurrentProbeSideWatermarkGauge().getValue()).isEqualTo(70L);
+ assertPhase(op, Phase.LOAD);
+
+ // Flip to JOIN.
+ addBuildWm(h, 1000L);
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(1000L);
+
+ // JOIN phase: probe WM gauge keeps tracking (guards the dedicated currentProbeWm
+ // field).
+ addProbeWm(h, 1500L);
+ assertThat(op.getCurrentProbeSideWatermarkGauge().getValue()).isEqualTo(1500L);
+ addBuildWm(h, 1100L);
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(1100L);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void fanOutGaugesTrackMaxAndAverage(boolean leftOuter) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(leftOuter, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // k1 multi-set: v1 x2, v2 x1 → fan-out 3; k2 single row → fan-out 1.
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v2", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ // a probe record during load
+ addProbeRecord(h, 1L, "k2", "p1"); // fan-out 1
+ // a probe record that does not match
+ addProbeRecord(h, 2L, "k3", "p2"); // fan-out INNER: 0, LEFT OUTER: 1
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // check merge stats after transition
+ assertThat(op.getMaxJoinFanOutGauge().getValue()).isEqualTo(1L);
+ if (leftOuter) {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isEqualTo(2d / 2);
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(0L);
+ } else {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(1d / 2, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(1L);
+ }
+
+ // join probe in JOIN phase
+ addProbeRecord(h, 3L, "k1", "p3"); // fan-out 3
+ assertThat(op.getMaxJoinFanOutGauge().getValue()).isEqualTo(3L);
+ if (leftOuter) {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isEqualTo(5d / 3, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(0L);
+ } else {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(4d / 3, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(1L);
+ }
+
+ addProbeRecord(h, 3L, "k3", "no-match");
+ assertThat(op.getMaxJoinFanOutGauge().getValue()).isEqualTo(3L);
+ if (leftOuter) {
+ // fan-out 1 (LEFT OUTER, null-padded)
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(6.0d / 4, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(0L);
+ } else {
+ // fan-out 0 (INNER, unmatched)
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(4.0d / 4, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(2L);
+ }
+ }
+ }
+
+ @Test
+ void numUnmatchedBuildRetractionsCountsAbsentRowRetractions() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // LOAD: a -D for a never-inserted row is counted.
+ addBuildChange(h, deleteRecord("k1", "ghost", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 2L));
+ assertThat(op.getNumUnmatchedBuildRetractions().getCount()).isEqualTo(0L);
+
+ // Add a real row for k1
+ addBuildChange(h, insertRecord("k1", "v1", 3L));
+ // Flip to JOIN
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // Access k1 → drains [-D(ghost)@1, +I(v1)@2]. The -D hits an absent row and is counted.
+ addBuildChange(h, deleteRecord("k1", "ghost2", 101L));
+ assertThat(op.getNumUnmatchedBuildRetractions().getCount()).isEqualTo(1L);
+
+ // Advance WM + access → drains -D(ghost2)@3 on an absent row.
+ addBuildWm(h, 110L);
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(op.getNumUnmatchedBuildRetractions().getCount()).isEqualTo(2L);
+
+ // Deleting an existing row (matching row-time) does not change the metric.
+ addBuildChange(h, deleteRecord("k1", "v1", 2L));
+ addBuildWm(h, 120L);
+ addProbeRecord(h, 2L, "k1", "p2");
+ assertThat(op.getNumUnmatchedBuildRetractions().getCount()).isEqualTo(2L);
+ }
+ }
+
+ @Test
+ void numStateTtlEvictionsCountsEvictedKeys() throws Exception {
+ // stateTtlMs = 100 → timers registered at 1.5 × = 150.
+ LateralSnapshotJoinOperator op = newOperator(false, 50L, null, 100L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ addBuildWm(h, 50L); // flip to JOIN at proc-time 0
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getNumStateTtlEvictions().getCount()).isEqualTo(0L);
+
+ // Neither key is touched again; both evict once their TTL timer (150) fires.
+ h.setProcessingTime(150);
+ assertThat(buildTableKeys(h)).isEmpty();
+ assertThat(op.getNumStateTtlEvictions().getCount()).isEqualTo(2L);
+ }
+ }
+
+ @Test
+ void bufferGaugesAreRebuiltFromStateOnRestore() throws Exception {
+ // Snapshot in LOAD with probes buffered across multiple keys, and in JOIN with build-side
+ // changes buffered across multiple keys, then assert both gauges are rebuilt from the
+ // restored keyed state (the in-memory tallies start at 0 in the restored operator).
+
+ // --- LOAD-phase snapshot: three buffered probes spread over two keys. ---
+ LateralSnapshotJoinOperator loadOp = newOperator(false, 100L, null, null);
+ OperatorSubtaskState loadState;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(loadOp)) {
+ h.open();
+ addProbeRecord(h, 1L, "k1", "p1");
+ addProbeRecord(h, 2L, "k2", "p2");
+ addProbeRecord(h, 3L, "k1", "p3");
+ assertPhase(loadOp, Phase.LOAD);
+ assertThat(loadOp.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+ loadState = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator loadOp2 = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(loadOp2)) {
+ h.initializeState(loadState);
+ h.open();
+ // Rebuilt from restored state: all three probes, across both keys.
+ assertThat(loadOp2.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+ // The build gauge stays 0 — nothing was buffered on the build side.
+ assertThat(loadOp2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ }
+
+ // --- JOIN-phase snapshot: three buffered build-side changes spread over two keys. ---
+ LateralSnapshotJoinOperator joinOp = newOperator(false, 100L, null, null);
+ OperatorSubtaskState joinState;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(joinOp)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(joinOp, Phase.JOIN);
+ // The v2 access drains the LOAD insert (v1); v2, v3 and k2/v1 remain buffered.
+ addBuildChange(h, insertRecord("k1", "v2", 101L));
+ addBuildChange(h, insertRecord("k1", "v3", 101L));
+ addBuildChange(h, insertRecord("k2", "v1", 101L));
+ assertThat(joinOp.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+ joinState = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator joinOp2 = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(joinOp2)) {
+ h.initializeState(joinState);
+ h.open();
+ assertPhase(joinOp2, Phase.JOIN);
+ // Rebuilt from restored state: all three buffered changes, across both keys.
+ assertThat(joinOp2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+ // The probe gauge stays 0 — nothing was buffered on the probe side.
+ assertThat(joinOp2.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ }
+ }
+
+ // ----------------------------------------------------------------- Helpers
+
+ /** Sends a build-side change record (any {@link RowKind}) to the harness. */
+ private static void addBuildChange(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h,
+ StreamRecord<RowData> record)
+ throws Exception {
+ h.processElement2(record);
+ }
+
+ /** Sends a probe-side {@link RowKind#INSERT} record with the default probe schema. */
+ private static void addProbeRecord(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h,
+ Long id,
+ String key,
+ String val)
+ throws Exception {
+ h.processElement1(insertRecord(id, key, val));
+ }
+
+ /** Sends a build-side watermark to the harness (input 2). */
+ private static void addBuildWm(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h, long ts)
+ throws Exception {
+ h.processWatermark2(new Watermark(ts));
+ }
+
+ /** Sends a probe-side watermark to the harness (input 1). */
+ private static void addProbeWm(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h, long ts)
+ throws Exception {
+ h.processWatermark1(new Watermark(ts));
+ }
+
+ /**
+ * Builds an expected joined row. The operator only ever emits {@link RowKind#INSERT}, so we
+ * don't accept a rowkind parameter.
+ */
+ private static GenericRowData row(
+ Long id, String pKey, String pVal, String bKey, String bVal, Long bRt) {
+ GenericRowData r = new GenericRowData(OUTPUT_TYPES.length);
+ r.setField(0, id);
+ r.setField(1, pKey == null ? null : StringData.fromString(pKey));
+ r.setField(2, pVal == null ? null : StringData.fromString(pVal));
+ r.setField(3, bKey == null ? null : StringData.fromString(bKey));
+ r.setField(4, bVal == null ? null : StringData.fromString(bVal));
+ r.setField(5, bRt);
+ r.setRowKind(RowKind.INSERT);
+ return r;
+ }
+
+ /**
+ * Builds an expected joined row for a composite (two-field) equi-key on both sides: probe
+ * {@code (kA, kB, val)} concatenated with build {@code (kA, kB, val)}.
+ */
+ private static GenericRowData compKeyRow(
+ String pKa, String pKb, String pVal, String bKa, String bKb, String bVal, Long bRt) {
+ GenericRowData r = new GenericRowData(7);
+ r.setField(0, pKa == null ? null : StringData.fromString(pKa));
+ r.setField(1, pKb == null ? null : StringData.fromString(pKb));
+ r.setField(2, pVal == null ? null : StringData.fromString(pVal));
+ r.setField(3, bKa == null ? null : StringData.fromString(bKa));
+ r.setField(4, bKb == null ? null : StringData.fromString(bKb));
+ r.setField(5, bVal == null ? null : StringData.fromString(bVal));
+ r.setField(6, bRt);
+ r.setRowKind(RowKind.INSERT);
+ return r;
+ }
+
+ private static void assertPhase(LateralSnapshotJoinOperator op, Phase expected) {
+ assertThat(op.getPhase()).isEqualTo(expected);
+ }
+
+ private static BinaryRowData stringKey(String key) {
+ BinaryRowData k = new BinaryRowData(1);
+ BinaryRowWriter w = new BinaryRowWriter(k);
+ if (key == null) {
+ w.setNullAt(0);
+ } else {
+ w.writeString(0, StringData.fromString(key));
+ }
+ w.complete();
+ return k;
+ }
+
+ private static List<String> buildTableKeys(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h)
+ throws Exception {
+ return h.getOperator()
+ .getKeyedStateBackend()
+ .getKeys(BUILD_TABLE_STATE_NAME, VoidNamespace.INSTANCE)
+ .map(r -> ((BinaryRowData) r).getString(0).toString())
+ .sorted()
+ .toList();
+ }
+
+ private static List<String> probeBufferKeys(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h)
+ throws Exception {
+ return h.getOperator()
+ .getKeyedStateBackend()
+ .getKeys(PROBE_BUFFER_STATE_NAME, VoidNamespace.INSTANCE)
+ .map(r -> ((BinaryRowData) r).getString(0).toString())
+ .sorted()
+ .toList();
+ }
+
+ /**
+ * Returns the keys that currently hold any build-side state — either a materialized build-table
+ * entry or a buffered (not-yet-applied) build-side change. TTL eviction clears both, so this
+ * reflects whether a key is still "alive", regardless of whether its buffered changes have been
+ * drained into the build table yet.
+ */
+ private static List<String> buildStateKeys(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h)
+ throws Exception {
+ java.util.SortedSet<String> keys = new java.util.TreeSet<>();
+ h.getOperator()
+ .getKeyedStateBackend()
+ .getKeys(BUILD_TABLE_STATE_NAME, VoidNamespace.INSTANCE)
+ .forEach(r -> keys.add(((BinaryRowData) r).getString(0).toString()));
+ h.getOperator()
+ .getKeyedStateBackend()
+ .getKeys(BUILD_CHANGE_BUFFER_STATE_NAME, VoidNamespace.INSTANCE)
+ .forEach(r -> keys.add(((BinaryRowData) r).getString(0).toString()));
+ return new ArrayList<>(keys);
+ }
+
+ /**
+ * Returns the build-table multi-set for the given key as {@code build-val → count}. Assumes the
+ * schema's value field is at index 1.
+ */
+ private static Map<String, Long> buildTableForKey(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h,
+ LateralSnapshotJoinOperator op,
+ String key)
+ throws Exception {
+ h.getOperator().setCurrentKey(stringKey(key));
+ Map<String, Long> result = new LinkedHashMap<>();
+ for (Map.Entry<RowData, Long> e : op.getBuildTableState().entries()) {
+ result.put(e.getKey().getString(1).toString(), e.getValue());
+ }
+ return result;
+ }
+
+ private static List<RowData> probeBufferForKey(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h,
+ LateralSnapshotJoinOperator op,
+ String key)
+ throws Exception {
+ h.getOperator().setCurrentKey(stringKey(key));
+ List<RowData> result = new ArrayList<>();
+ for (RowData r : op.getProbeBuffer().get()) {
+ result.add(r);
+ }
+ return result;
+ }
+
+ private static List<RowData> bufferedChangesForKey(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h,
+ LateralSnapshotJoinOperator op,
+ String key)
+ throws Exception {
+ h.getOperator().setCurrentKey(stringKey(key));
+ List<RowData> result = new ArrayList<>();
+ for (RowData r : op.getBuildChangeBuffer().get()) {
+ result.add(r);
+ }
+ return result;
+ }
+
+ private static Long bufferedAtWmFor(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h,
+ LateralSnapshotJoinOperator op,
+ String key)
+ throws Exception {
+ h.getOperator().setCurrentKey(stringKey(key));
+ return op.getBufferedAtWmState().value();
+ }
+
+ private static Long ttlExpiryFor(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h,
+ LateralSnapshotJoinOperator op,
+ String key)
+ throws Exception {
+ h.getOperator().setCurrentKey(stringKey(key));
+ return op.getTtlExpiryState().value();
+ }
+
+ /** Drops watermarks and watermark statuses from the harness output queue, in place. */
+ private static void stripWatermarksAndStatusesFromOutput(
+ KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h) {
+ h.getOutput().removeIf(o -> o instanceof Watermark || o instanceof WatermarkStatus);
+ }
+
+ private static List<Object> stripWatermarks(ConcurrentLinkedQueue<Object> output) {
Review Comment:
I think `stripWatermarks(...)` ends up unused here - only `stripWatermarksAndStatusesFromOutput` / `extractWatermarks` are called - safe to delete?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3461343173
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
Review Comment:
The idle timer is (re-)started in `open()` after state recovery, so this time should not reduce the effective idle interval.
Wouldn't a new build-side record also cause the emission of a new build-side WM due to periodic watermark generation (default being 200ms)?
Or am I misunderstanding your point?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3500627541
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
Review Comment:
OK, sounds good. I'll change the approach to fine-grained timers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske merged PR #28329:
URL: https://github.com/apache/flink/pull/28329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3572868866
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
Review Comment:
Thanks, that resolves my concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3491142761
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
Review Comment:
The `bufferedAtWmState` is only valid from the time of buffering a build-side record until the buffer is applied. Afterwards it is reset.
In case of a recovery, it might be set to `MIN_VALUE` if we didn't process a build-side WM before.
This is not a problem because we only use it to figure out if we received a WM after we started to buffer to ensure that updates are atomically applied.
The value is not emitted downstream as a watermark.
I've added some comments to explain the logic.
Let me know if it needs further explanation.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
Review Comment:
The `bufferedAtWmState` is only valid from the time of buffering a build-side record until the buffer is applied. Afterwards it is reset.
In case of a recovery, it might be set to `MIN_VALUE` if we didn't process a build-side WM before.
This is not a problem because we only use it to figure out if we received a WM after we started to buffer to ensure that updates are atomically applied.
The value is not emitted downstream as a watermark.
I've added some comments to explain the logic.
Please let me know if it needs further explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3483285170
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
Review Comment:
But can they detect that the **joined** record is late?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3476363180
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
Review Comment:
That's a very good point!
I've replaced `probeBuffer` which might store many records with a `MapState`.
I'm not using the entry iterator to ensure that the buffered rows are read in the order in which they were inserted. RocksDBStateBackend would give us that order (due to the encoding of the long index values) but HeapStateBackend uses HashMap under the hood which doesn't preserve the order.
`buildChangeBuffer` should only buffer very little data (only changes between two WMs to ensure atomic application). So this should be fine, IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3491226873
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
Review Comment:
There are two (disjunctive) conditions to flip from LOAD to JOIN.
Both are triggered out-side of a keyed context:
1) the build-side WM advances past a configured timestamp
2) the operator processing-time idle timer fires
In both cases, we pass the `currentProbeSideWm` that we remembered to the timer service, which triggers the keyed event-time timers that drain the buffered probe-side records (in an interruptible fashion since the PR update).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3436591190
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
Review Comment:
I'm +1 for not using abbreviations. I could figure out most not sure what "NS" means here. Namespace? But yeah, it'll be easier for human reviewers with less context and even AI
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
Review Comment:
Can we use ListState<Phase>?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3554727601
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
Review Comment:
(was fixed in `[FLINK-39781][table] Gate buffered build-change application on row-time` by stopping at `currentBuildSideWm`)
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
Review Comment:
Thanks!
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
Review Comment:
Thank you!
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1078 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.EnumSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.StateBackendLoader;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffer is keyed by the
+ * build-side row-time attribute ({@code buildRowtimeIndex}) and applied in row-time order
+ * once the build watermark reaches it; changes sharing a row-time are applied in arrival
+ * order, which mirrors the upstream emit order (a retraction directly precedes its
+ * accumulation). Gating application on the row-time preserves atomic update visibility across
+ * {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD each is
+ * buffered in {@code probeBuffer} under a synthetic, increasing timestamp with one event-time
+ * timer registered per record. At the flip these timers fire (one record each, interruptibly)
+ * and join the buffered probes against the materialized build-side state in insertion order.
+ * During JOIN records are joined immediately, unless the key's flip drain has not finished
+ * yet, in which case the record is buffered behind the pending ones to preserve order.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ *
+ * <p>Streaming only: The operator joins data with processing-time semantics which can't be (easily)
+ * done in batch. Also, the operator's phase is held in union operator state, which is incompatible
+ * with finished operators. Joins against LATERAL SNAPSHOT functions should be translated to regular
+ * joins. The changes on the build-side would be consolidated into a final table and then be joined
+ * with the probe-side. All probe-side records are then joined against the same (and final) version
+ * of the build-side input.
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>,
+ Triggerable<RowData, VoidNamespace> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** Operator state names. */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** Keyed state names. */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String PROBE_BUFFER_SEQ_STATE_NAME = "probe-buffer-seq";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /**
+ * Single timer service for two kinds of per-key timers, distinguished by timer type (not by
+ * namespace): event-time timers drain a key's LOAD-buffered probes at the flip ({@link
+ * #onEventTime}), processing-time timers evict keyed state ({@link #onProcessingTime}). Both
+ * use {@link VoidNamespace} so no namespace is serialized per timer.
+ */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting
+ static final String NUM_PROBE_BUFFERED_METRIC_NAME = "numProbeSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting
+ static final String NUM_STATE_TTL_EVICTIONS_METRIC_NAME = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_BUILD_WM_METRIC_NAME = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_PROBE_WM_METRIC_NAME = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String CURRENT_PHASE_METRIC_NAME = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String MAX_JOIN_FAN_OUT_METRIC_NAME = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String AVG_JOIN_FAN_OUT_METRIC_NAME = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_PROBE_METRIC_NAME = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME =
+ "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<VoidNamespace> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ /**
+ * True if the keyed state backend iterates map entries in serialized-key order (RocksDB/ForSt),
+ * so the row-time-keyed {@link #buildChangeBuffer} is already scanned in event-time order.
+ */
+ private transient boolean sortedStateBackend;
+
+ // -------------------------- keyed state --------------------------
+
+ /**
+ * Build-side table as multi-set (row → reference count). Holds the committed snapshot, i.e. all
+ * build changes up to the last applied build watermark; probes join against this.
+ */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /**
+ * Build-side changes not yet visible in {@link #buildTableState}, keyed by their build row-time
+ * (multiple changes with the same row-time share a list value). Changes are applied to {@code
+ * buildTableState} once the build watermark reaches their row-time.
+ */
+ private transient MapState<Long, List<RowData>> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /**
+ * Buffer for probe-side records during LOAD (and for the rare JOIN-phase record that arrives
+ * while a key's buffer is still draining). Keyed by a synthetic, monotonically increasing
+ * timestamp so the per-record flip timers fire in insertion order. Each buffered row has one
+ * event-time timer registered on its key; the timer drains exactly that row.
+ */
+ private transient MapState<Long, RowData> probeBuffer;
+
+ /**
+ * Next sequence number for the current key, used to derive distinct, ordered synthetic
+ * timestamps for {@link #probeBuffer}. {@code null} iff the buffer is empty, so it doubles as a
+ * cheap "buffer non-empty" flag on the JOIN-phase fast path.
+ */
+ private transient ValueState<Long> probeBufferSeq;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<Phase> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ if (buildRowtimeIndex >= rightType.toRowType().getFieldCount()) {
+ throw new IllegalArgumentException("buildRowtimeIndex out of bounds for build row");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME,
+ new EnumSerializer<>(Phase.class)));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (Phase persistedPhase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (persistedPhase == Phase.LOAD) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ probeBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME,
+ Types.LONG,
+ Types.LIST(rightType)));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ PROBE_BUFFER_STATE_NAME, Types.LONG, leftType));
+ probeBufferSeq =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(
+ PROBE_BUFFER_SEQ_STATE_NAME, Types.LONG));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice. Both timer kinds use VoidNamespace; they are told apart by timer type
+ // (event-time -> flip drain, processing-time -> TTL eviction), not by namespace.
+ timerService =
+ getInternalTimerService(TIMER_SERVICE_NAME, VoidNamespaceSerializer.INSTANCE, this);
+
+ // RocksDB/ForSt iterate map entries in serialized-key order, so the row-time-keyed build
+ // change buffer is scanned in event-time order; unordered backends (heap) need a sort.
+ final String backendId = getKeyedStateBackend().getBackendTypeIdentifier();
+ sortedStateBackend =
+ StateBackendLoader.ROCKSDB_STATE_BACKEND_NAME.equals(backendId)
+ || StateBackendLoader.FORST_STATE_BACKEND_NAME.equals(backendId);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(NUM_STATE_TTL_EVICTIONS_METRIC_NAME);
+ numUnmatchedProbeRecords = metricGroup.counter(NUM_UNMATCHED_PROBE_METRIC_NAME);
+ numUnmatchedBuildRetractions =
+ metricGroup.counter(NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME);
+
+ // Best-effort in-memory tally; not restored, so it starts at 0 after a restore/rescale.
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(NUM_PROBE_BUFFERED_METRIC_NAME, probeBufferedGauge);
+ metricGroup.gauge(CURRENT_BUILD_WM_METRIC_NAME, buildWmGauge);
+ metricGroup.gauge(CURRENT_PROBE_WM_METRIC_NAME, probeWmGauge);
+ metricGroup.gauge(MAX_JOIN_FAN_OUT_METRIC_NAME, maxFanOutGauge);
+ metricGroup.gauge(AVG_JOIN_FAN_OUT_METRIC_NAME, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(CURRENT_PHASE_METRIC_NAME, phaseGauge);
+
+ // Pin the build-side input idle: the operator absorbs build-side watermarks and status.
+ combinedWatermark.updateStatus(1, true);
+
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ // Buffer during LOAD; in JOIN, buffer too if this key's flip drain has not finished yet
+ // (probeBufferSeq != null), so the new probe is joined in order after the buffered ones by
+ // its own timer instead of draining the whole buffer inline. Otherwise join immediately.
+ if (phase == Phase.LOAD || probeBufferSeq.value() != null) {
+ bufferProbe(probe);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ /**
+ * Buffers a probe row for the current key and registers a per-record event-time flip timer. The
+ * synthetic timestamp {@code (seq + 1) - Long.MAX_VALUE} is negative (so any non-negative
+ * watermark fires it) and increases with the sequence number (so timers drain in insertion
+ * order on every state backend).
+ */
+ private void bufferProbe(RowData probe) throws Exception {
+ long seq = probeBufferSeq.value() == null ? 0L : probeBufferSeq.value();
+ long ts = (seq + 1) - Long.MAX_VALUE;
+ probeBuffer.put(ts, probe);
+ probeBufferSeq.update(seq + 1);
+ timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, ts);
+ probeBufferedGauge.update(++probeBufferedCount);
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the change under its build row-time (grouping changes that share a row-time).
+ long rowtime = build.getLong(buildRowtimeIndex);
+ List<RowData> atRowtime = buildChangeBuffer.get(rowtime);
+ if (atRowtime == null) {
+ atRowtime = new ArrayList<>();
+ }
+ atRowtime.add(build);
+ buildChangeBuffer.put(rowtime, atRowtime);
+ // Tag the buffer with the current build watermark so it drains once the watermark passes.
+ // The tag may move backwards on restore (currentBuildSideWm resets to MIN_VALUE), which
+ // re-anchors a recovered buffer to this subtask's watermark.
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed; combined watermark is probe-driven.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // LOAD emits nothing downstream; track partial[0]'s idle bit for use after the flip.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only event-time (flip) timers are registered: each drains exactly one buffered probe for
+ // its key. All of a key's timers become due at the flip; interruptible timers fire them one
+ // at a time, yielding to checkpoints between firings.
+ long ts = timer.getTimestamp();
+ // Apply this key's due build-side changes before joining, so probes see the materialized
+ // build snapshot. Idempotent across the key's firings within one watermark sweep.
+ applyDueBufferedChanges();
+ RowData probe = probeBuffer.get(ts);
+ if (probe == null) {
+ // Already drained or evicted (e.g. by TTL) before this timer fired.
+ return;
+ }
+ joinProbeRow(probe);
+ probeBuffer.remove(ts);
+ probeBufferedCount = Math.max(0, probeBufferedCount - 1);
+ probeBufferedGauge.update(probeBufferedCount);
+ // This is the last buffered row iff it carries the highest sequence number; clearing the
+ // sequence here marks the buffer empty (cheap point lookup, no isEmpty() scan).
+ Long seq = probeBufferSeq.value();
+ if (seq != null && ts + Long.MAX_VALUE == seq) {
+ probeBufferSeq.clear();
+ }
+ }
+
+ /** Removes all probe-buffer state for the current key. */
+ private void clearProbeBuffer() {
+ probeBuffer.clear();
+ probeBufferSeq.clear();
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only processing-time (TTL) timers are registered. Semantics match Flink's standard
+ // StateTtlConfig.
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // Reschedule if still in LOAD, or in JOIN but within stateTtlMs of the flip (grace window).
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(VoidNamespace.INSTANCE, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ clearAllPerKeyState();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN. Runs in a NON-KEYED context (callers {@link #processWatermark2}
+ * and {@link #idleFlipTimer}), so it must not touch keyed state directly; the per-key probe
+ * drain happens in the per-record event-time timers fired below.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Anchor the TTL grace window at the flip so keys loaded before it aren't evicted early.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ cancelIdleFlipTimer();
+
+ // If we have seen a (non-negative) probe-side WM, forward it so the per-record flip timers
+ // fire (their synthetic timestamps are negative) and drain the buffered probes. Firing is
+ // interruptible and the watermark is forwarded after the buffered probes are joined.
+ // If we haven't seen a probe-side WM yet, draining is triggered by the next probe-side WM.
+ if (currentProbeSideWm >= 0) {
+ super.processWatermark1(new Watermark(currentProbeSideWm));
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm >= 0 ? currentProbeSideWm : "none");
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // The build-side wm advanced: apply the changes that are now due (their row-time has
+ // been reached by the build-side watermark) and keep any not-yet-due changes buffered.
+ applyDueBufferedChanges();
+ return bufferedAtWmState.value();
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark seen by this subtask (recovery/rescale,
+ // or idle-timeout flip). Force-apply everything now, since there is no watermark to
+ // gate on and the next build watermark may not arrive for a long time.
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Applies the buffered build-side changes that are due — i.e. whose row-time has been reached
+ * by the build-side watermark — in event-time order, leaving any not-yet-due changes untouched
+ * in state.
+ */
+ private void applyDueBufferedChanges() throws Exception {
Review Comment:
Thanks!
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
+ buildTableState.put(change, currentCnt == null ? 1L : currentCnt + 1L);
+ } else {
+ // -D / -U
+ if (currentCnt == null || currentCnt <= 0L) {
+ numUnmatchedBuildRetractions.inc();
+ return;
+ }
Review Comment:
I think now that we're gating the changes-to-apply by watermark; it's a more explicit contract with the user/source that all the related changes show up until that WM.
One discrepancy though is that we keep late INSERTs but drop late DELETEs (I'm referring to 0000 comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3572549131
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
Review Comment:
Ah, sorry for missing this comment.
Also I didn't understand Piotr's point correctly initially.
You're both right, starting the timer in `open()` is risky.
I've changed the implementation to start the timer on the first build-side activity (record, wm, or wm status).
@rkhachatryan Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3504208861
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
Review Comment:
I've removed the timer namespace to reduce the timer serialization overhead (we're creating a lot more timer now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3453697545
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
Review Comment:
The later state clean up doesn't touch all keyed states.
We can add a helper method, but IMO it's not going to make the code easier to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3483178801
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
Review Comment:
> Buffering build-side changes prevents that we join against incomplete update changes (only -U/-D have been processed, and +U/+I haven't arrived yet).
But that's exactly what we do here, don't we?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
Review Comment:
I probably misunderstood this part, but then why do we need buffering after flip?
If that's to prevent out-of-order changelog events (I guess) then we're losing that protection?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
Review Comment:
Could you elaborate why deletion comes before insertion?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
Review Comment:
I think it's the key for one side but not the other? (we can't have both)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3500638174
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
+ buildTableState.put(change, currentCnt == null ? 1L : currentCnt + 1L);
+ } else {
+ // -D / -U
+ if (currentCnt == null || currentCnt <= 0L) {
+ numUnmatchedBuildRetractions.inc();
+ return;
+ }
Review Comment:
Sorry, I might have misread your comment.
Are you concerned that we are dropping out-of-order records or wondering why we aren't dropping them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3437217142
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
Review Comment:
Tiny thing: `Phase` is `public` but I don't think anything outside needs it (planner doesn't, tests share the package) - maybe make it package-private? Or do you already have plans for other users
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3453820849
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
+ buildTableState.put(change, currentCnt == null ? 1L : currentCnt + 1L);
+ } else {
+ // -D / -U
+ if (currentCnt == null || currentCnt <= 0L) {
+ numUnmatchedBuildRetractions.inc();
+ return;
+ }
Review Comment:
Hmm, good point. I'll have to think a bit about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3545710161
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1078 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.EnumSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.StateBackendLoader;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffer is keyed by the
+ * build-side row-time attribute ({@code buildRowtimeIndex}) and applied in row-time order
+ * once the build watermark reaches it; changes sharing a row-time are applied in arrival
+ * order, which mirrors the upstream emit order (a retraction directly precedes its
+ * accumulation). Gating application on the row-time preserves atomic update visibility across
+ * {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD each is
+ * buffered in {@code probeBuffer} under a synthetic, increasing timestamp with one event-time
+ * timer registered per record. At the flip these timers fire (one record each, interruptibly)
+ * and join the buffered probes against the materialized build-side state in insertion order.
+ * During JOIN records are joined immediately, unless the key's flip drain has not finished
+ * yet, in which case the record is buffered behind the pending ones to preserve order.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ *
+ * <p>Streaming only: The operator joins data with processing-time semantics which can't be (easily)
+ * done in batch. Also, the operator's phase is held in union operator state, which is incompatible
+ * with finished operators. Joins against LATERAL SNAPSHOT functions should be translated to regular
+ * joins. The changes on the build-side would be consolidated into a final table and then be joined
+ * with the probe-side. All probe-side records are then joined against the same (and final) version
+ * of the build-side input.
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>,
+ Triggerable<RowData, VoidNamespace> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** Operator state names. */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** Keyed state names. */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String PROBE_BUFFER_SEQ_STATE_NAME = "probe-buffer-seq";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /**
+ * Single timer service for two kinds of per-key timers, distinguished by timer type (not by
+ * namespace): event-time timers drain a key's LOAD-buffered probes at the flip ({@link
+ * #onEventTime}), processing-time timers evict keyed state ({@link #onProcessingTime}). Both
+ * use {@link VoidNamespace} so no namespace is serialized per timer.
+ */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting
+ static final String NUM_PROBE_BUFFERED_METRIC_NAME = "numProbeSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting
+ static final String NUM_STATE_TTL_EVICTIONS_METRIC_NAME = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_BUILD_WM_METRIC_NAME = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_PROBE_WM_METRIC_NAME = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String CURRENT_PHASE_METRIC_NAME = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String MAX_JOIN_FAN_OUT_METRIC_NAME = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String AVG_JOIN_FAN_OUT_METRIC_NAME = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_PROBE_METRIC_NAME = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME =
+ "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<VoidNamespace> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ /**
+ * True if the keyed state backend iterates map entries in serialized-key order (RocksDB/ForSt),
+ * so the row-time-keyed {@link #buildChangeBuffer} is already scanned in event-time order.
+ */
+ private transient boolean sortedStateBackend;
+
+ // -------------------------- keyed state --------------------------
+
+ /**
+ * Build-side table as multi-set (row → reference count). Holds the committed snapshot, i.e. all
+ * build changes up to the last applied build watermark; probes join against this.
+ */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /**
+ * Build-side changes not yet visible in {@link #buildTableState}, keyed by their build row-time
+ * (multiple changes with the same row-time share a list value). Changes are applied to {@code
+ * buildTableState} once the build watermark reaches their row-time.
+ */
+ private transient MapState<Long, List<RowData>> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /**
+ * Buffer for probe-side records during LOAD (and for the rare JOIN-phase record that arrives
+ * while a key's buffer is still draining). Keyed by a synthetic, monotonically increasing
+ * timestamp so the per-record flip timers fire in insertion order. Each buffered row has one
+ * event-time timer registered on its key; the timer drains exactly that row.
+ */
+ private transient MapState<Long, RowData> probeBuffer;
+
+ /**
+ * Next sequence number for the current key, used to derive distinct, ordered synthetic
+ * timestamps for {@link #probeBuffer}. {@code null} iff the buffer is empty, so it doubles as a
+ * cheap "buffer non-empty" flag on the JOIN-phase fast path.
+ */
+ private transient ValueState<Long> probeBufferSeq;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<Phase> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
Review Comment:
You changes the field to the long primitive ins the class long. Small follow-up: the constructor param is still Long loadCompletedTime. Since it's mandatory (checkNotNull, no @Nullable), a primitive long here would match the field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3554830368
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
Review Comment:
Ok, thanks for clarifying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3554944214
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
Review Comment:
I think @pnowojski concern is that the processing time might advance before any input is received
And the idea is to solve by`rescheduleIdleFlipTimer`, but probably it won't always work:
1. There are two or more`LateralSnapshotJoinOperator` in the pipeline. The 2nd operator might timeout because it won't see anything for a long time
2. Unaligned checkpoint recovery with heavy back-pressure - preventing **_new_** WM propagation - persisted build-side WMs (in channel state) should help here
3. Same, but with too many probe-side records in the unaligned checkpoint (and no persisted build-side WMs)
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
Review Comment:
I think it should work as long as WM is updated after firing all the timers - probe-side case.
On build-side, however, the WM is updated before firing the timers:
```
@Override
public void processWatermark2(Watermark mark) throws Exception {
// Build-side watermark: NEVER forwarded; never advances the timer service.
long ts = mark.getTimestamp();
currentBuildSideWm = Math.max(currentBuildSideWm, ts); <-- update WM
buildWmGauge.update(currentBuildSideWm);
if (phase == Phase.LOAD) {
if (currentBuildSideWm >= loadCompletedTime) {
// we reached the flip point. Transition to JOIN phase.
transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK); <--- fire timers
} else if (loadCompletedIdleTimeoutMs != null) {
// we got a new build-side wm. Reschedule the idle timer (if it was configured)
rescheduleIdleFlipTimer();
}
}
}
```
I don't see anything wrong with `currentBuildSideWm`, do you?
Probably makes sense to add test coverage for this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3444161268
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
Review Comment:
Flink has a limitation that union state is [not compatible](https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/dev/datastream/fault-tolerance/checkpointing/#how-does-this-impact-the-operator-state) with finished operators with bounded jobs.
That means that bounded jobs using this operator might enter restart loop / fail depending on recovery strategy and parallelism.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
Review Comment:
Don't we want to drop late records here?
I'm not sure if downstream operators can reliably detect lateness after joining.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
Review Comment:
1. This can flip `bufferedAtWmState` to `MIN_VALUE` or make it go backwards depending on the return value of `applyBufferedChangesIfReady`.
Currently, this is not possible but such a dependency seems fragile.
2. Add a comment about updating key-scoped state from subtask-scoped variable? To me, this logic is not obvious.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
Review Comment:
ListState.get loads the whole state (for the current key) into memory, which may lead to OOM.
Can/should we avoid that?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
Review Comment:
Why not just call super? This call skips interruptible timers feature making the transition a single non-interruptible computation (i.e. checkpoints will have to wait).
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
Review Comment:
Can we break the iteration early when joining against PK?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
Review Comment:
Again, this loads the whole buffer into memory.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
Review Comment:
When called from `processElement1` in join phase, `MapState buildTableState` contains intermediate results (i.e. not "squashed by watermark").
Is it correct to join against them?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
Review Comment:
Is it correct to join against the "future" build-side records, i.e. the records with timestamp greater than combined watermark?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
+ buildTableState.put(change, currentCnt == null ? 1L : currentCnt + 1L);
+ } else {
+ // -D / -U
+ if (currentCnt == null || currentCnt <= 0L) {
+ numUnmatchedBuildRetractions.inc();
+ return;
+ }
Review Comment:
Aren't we dropping "user-caused" out-of-order -D/-U records here?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
Review Comment:
Why do we schedule the timer for time=1 and then match by namespace - is it to make the transition "atomic"?
Performance-wise, individual (splittable) timers would work much better:
We could have `MapState<timestamp, buffered prob side>` with a single map.remove on event time.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
Review Comment:
I'm confused about the order here:
- `-U` followed by `+U` makes sense
- But `-D` followed by `+I` seems a mistake
Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3436446730
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
Review Comment:
Can we use `ListState<Phase>`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3476367131
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
Review Comment:
Please see my reply to your comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3492522410
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
Review Comment:
I meant [this one](https://github.com/apache/flink/pull/28329#discussion_r3476363180).
`buildChangeBuffer` is drained on each key access (`processElement1()` or `processElement2()`) after the build-side WM advanced. The expectation is that whenever we received a build-side record, it won't take long until we receive a watermark. This bounds the number of records that are buffered in `buildChangeBuffer` to the number of records that are received between two watermarks (partitioned across all keys).
Hence, I expect each list to remain small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3491190900
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
Review Comment:
The idea is to not advance the (internal and downstream) WM at all during the LOAD phase. Instead we track the progress of the probe-side WM in `currentProbeSideWm` and use it to fire the event-time timers once the flip condition is met.
I've reworked the logic of the timers that fire during the transition from LOAD to JOIN.
First, I'm forwarding the WM to `super.processWatermark1()` to use the interruptible timer processing path. Second, the probe-side records are processed in chunks of (currently) 128 records. If the chunk size is exceeded, the timer logic stops, updates an offset and registers another timer on the same time. That should ensure that the flip timer logic can be interrupted by checkpoints.
Please let me know if this addresses your concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3491190900
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
Review Comment:
The idea is to not advance the (internal and downstream) WM at all during the LOAD phase. Instead we track the progress of the probe-side WM in `currentProbeSideWm` and use it to fire the event-time timers once the flip condition is met.
I've reworked the logic of the timers that fire during the transition from LOAD to JOIN.
First, I'm forwarding the WM to `super.processWatermark1()` to use the interruptible timer processing path. Second, the probe-side records are processed by individual timers so that the flip processing can be quickly interrupted.
Please let me know if this addresses your concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3545710161
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1078 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.EnumSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.StateBackendLoader;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffer is keyed by the
+ * build-side row-time attribute ({@code buildRowtimeIndex}) and applied in row-time order
+ * once the build watermark reaches it; changes sharing a row-time are applied in arrival
+ * order, which mirrors the upstream emit order (a retraction directly precedes its
+ * accumulation). Gating application on the row-time preserves atomic update visibility across
+ * {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD each is
+ * buffered in {@code probeBuffer} under a synthetic, increasing timestamp with one event-time
+ * timer registered per record. At the flip these timers fire (one record each, interruptibly)
+ * and join the buffered probes against the materialized build-side state in insertion order.
+ * During JOIN records are joined immediately, unless the key's flip drain has not finished
+ * yet, in which case the record is buffered behind the pending ones to preserve order.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ *
+ * <p>Streaming only: The operator joins data with processing-time semantics which can't be (easily)
+ * done in batch. Also, the operator's phase is held in union operator state, which is incompatible
+ * with finished operators. Joins against LATERAL SNAPSHOT functions should be translated to regular
+ * joins. The changes on the build-side would be consolidated into a final table and then be joined
+ * with the probe-side. All probe-side records are then joined against the same (and final) version
+ * of the build-side input.
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>,
+ Triggerable<RowData, VoidNamespace> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** Operator state names. */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** Keyed state names. */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String PROBE_BUFFER_SEQ_STATE_NAME = "probe-buffer-seq";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /**
+ * Single timer service for two kinds of per-key timers, distinguished by timer type (not by
+ * namespace): event-time timers drain a key's LOAD-buffered probes at the flip ({@link
+ * #onEventTime}), processing-time timers evict keyed state ({@link #onProcessingTime}). Both
+ * use {@link VoidNamespace} so no namespace is serialized per timer.
+ */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting
+ static final String NUM_PROBE_BUFFERED_METRIC_NAME = "numProbeSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting
+ static final String NUM_STATE_TTL_EVICTIONS_METRIC_NAME = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_BUILD_WM_METRIC_NAME = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_PROBE_WM_METRIC_NAME = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String CURRENT_PHASE_METRIC_NAME = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String MAX_JOIN_FAN_OUT_METRIC_NAME = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String AVG_JOIN_FAN_OUT_METRIC_NAME = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_PROBE_METRIC_NAME = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME =
+ "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<VoidNamespace> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ /**
+ * True if the keyed state backend iterates map entries in serialized-key order (RocksDB/ForSt),
+ * so the row-time-keyed {@link #buildChangeBuffer} is already scanned in event-time order.
+ */
+ private transient boolean sortedStateBackend;
+
+ // -------------------------- keyed state --------------------------
+
+ /**
+ * Build-side table as multi-set (row → reference count). Holds the committed snapshot, i.e. all
+ * build changes up to the last applied build watermark; probes join against this.
+ */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /**
+ * Build-side changes not yet visible in {@link #buildTableState}, keyed by their build row-time
+ * (multiple changes with the same row-time share a list value). Changes are applied to {@code
+ * buildTableState} once the build watermark reaches their row-time.
+ */
+ private transient MapState<Long, List<RowData>> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /**
+ * Buffer for probe-side records during LOAD (and for the rare JOIN-phase record that arrives
+ * while a key's buffer is still draining). Keyed by a synthetic, monotonically increasing
+ * timestamp so the per-record flip timers fire in insertion order. Each buffered row has one
+ * event-time timer registered on its key; the timer drains exactly that row.
+ */
+ private transient MapState<Long, RowData> probeBuffer;
+
+ /**
+ * Next sequence number for the current key, used to derive distinct, ordered synthetic
+ * timestamps for {@link #probeBuffer}. {@code null} iff the buffer is empty, so it doubles as a
+ * cheap "buffer non-empty" flag on the JOIN-phase fast path.
+ */
+ private transient ValueState<Long> probeBufferSeq;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<Phase> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
Review Comment:
You changes the field to the long primitive ins the class long. Small follow-up: the constructor param is still Long loadCompletedTime. Since it's mandatory (checkNotNull, no @Nullable), a primitive long here would match the field
```suggestion
long loadCompletedTime,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3481055379
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperatorTest.java:
##########
@@ -0,0 +1,1666 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.KeyedTwoInputStreamOperatorTestHarness;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.data.writer.BinaryRowWriter;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.Phase;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.runtime.util.RowDataHarnessAssertor;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.flink.types.RowKind;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_CHANGE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_TABLE_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.PROBE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.deleteRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.insertRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateAfterRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateBeforeRecord;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.within;
+
+/** Harness tests for {@link LateralSnapshotJoinOperator}. */
+class LateralSnapshotJoinOperatorTest {
+
+ // ----------------------------------------------------------------- Schema
+
+ /** Probe row schema: (id BIGINT, key VARCHAR, val VARCHAR). */
+ private static final InternalTypeInfo<RowData> PROBE_TYPE =
+ InternalTypeInfo.ofFields(
+ new BigIntType(), VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+
+ /** Build row schema: (key VARCHAR, val VARCHAR, rt BIGINT). */
+ private static final InternalTypeInfo<RowData> BUILD_TYPE =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE, VarCharType.STRING_TYPE, new BigIntType());
+
+ /** Joined output schema: probe ++ build = (id, pKey, pVal, bKey, bVal, bRt). */
+ private static final LogicalType[] OUTPUT_TYPES = {
+ new BigIntType(),
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType()
+ };
+
+ /** Probe key column index (key VARCHAR is at field 1). */
+ private static final int PROBE_KEY_IDX = 1;
+
+ /** Build key column index (key VARCHAR is at field 0). */
+ private static final int BUILD_KEY_IDX = 0;
+
+ /** Build row-time column index (rt BIGINT is at field 2). */
+ private static final int BUILD_RT_IDX = 2;
+
+ private static final InternalTypeInfo<RowData> KEY_TYPE =
+ InternalTypeInfo.ofFields(VarCharType.STRING_TYPE);
+
+ private static final KeySelector<RowData, RowData> PROBE_KEY_SELECTOR =
+ nullSafeStringKeySelector(PROBE_KEY_IDX);
+ private static final KeySelector<RowData, RowData> BUILD_KEY_SELECTOR =
+ nullSafeStringKeySelector(BUILD_KEY_IDX);
+
+ private static final RowDataHarnessAssertor JOINED_ASSERTOR =
+ new RowDataHarnessAssertor(OUTPUT_TYPES);
+
+ // ----------------------------------------------------------------- Join conditions
+
+ /** Trivial join condition that always matches (equality is enforced by partitioning). */
+ private static final String ALWAYS_TRUE_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionStub extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionStub(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) { return true; }\n"
+ + "}\n";
+
+ /**
+ * Join condition that only matches when the probe value (field 2) equals {@code "match"}. Used
+ * to verify that the codegen'd condition is actually invoked at join time.
+ */
+ private static final String MATCH_VAL_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionMatchVal extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionMatchVal(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) {\n"
+ + " if (in1.isNullAt(2)) { return false; }\n"
+ + " return \"match\".equals(in1.getString(2).toString());\n"
+ + " }\n"
+ + "}\n";
+
+ private static GeneratedJoinCondition newTrueCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionStub", ALWAYS_TRUE_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ private static GeneratedJoinCondition newMatchValCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionMatchVal", MATCH_VAL_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ // ----------------------------------------------------------------- Operator / harness
+ // factories
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ GeneratedJoinCondition joinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return new LateralSnapshotJoinOperator(
+ isLeftOuterJoin,
+ PROBE_TYPE,
+ BUILD_TYPE,
+ BUILD_RT_IDX,
+ joinCondition,
+ filterNullKeys,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return newOperator(
+ isLeftOuterJoin,
+ newTrueCondition(),
+ new boolean[] {true},
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData>
+ newHarness(LateralSnapshotJoinOperator op) throws Exception {
+ return new KeyedTwoInputStreamOperatorTestHarness<>(
+ op, PROBE_KEY_SELECTOR, BUILD_KEY_SELECTOR, KEY_TYPE);
+ }
+
+ private static KeySelector<RowData, RowData> nullSafeStringKeySelector(final int keyIdx) {
+ return value -> {
+ BinaryRowData ret = new BinaryRowData(1);
+ BinaryRowWriter writer = new BinaryRowWriter(ret);
+ if (value.isNullAt(keyIdx)) {
+ writer.setNullAt(0);
+ } else {
+ writer.writeString(0, value.getString(keyIdx));
+ }
+ writer.complete();
+ return ret;
+ };
+ }
+
+ // ----------------------------------------------------------------- LOAD phase
+
+ @Test
+ void loadPhaseBuildSideChangeProcessing() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // During LOAD, build-side changes are buffered and later applied in event-time order.
+ // -D for a never-inserted (key, value) pair is defensively ignored.
+ addBuildChange(h, deleteRecord("k1", "ghost", 5L));
+ // Two identical records (same key/val/row-time) → count(k1, v1, 20) = 2.
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ // Earlier row-time than v1, but arrives later.
+ addBuildChange(h, insertRecord("k1", "v2", 10L));
+
+ // Still LOAD: changes are buffered, nothing applied or emitted yet.
+ assertPhase(op, Phase.LOAD);
+ assertThat(h.getOutput()).isEmpty();
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(buildTableKeys(h)).isEmpty();
+
+ // Advance the build watermark (still below the flip point) and access k1 again. The
+ // access drains the buffered batch in event-time order before buffering the new change.
+ addBuildWm(h, 50L);
+ addBuildChange(h, insertRecord("k1", "v3", 30L));
+ // TODO: also add updateBefore and updateAfter changes, to ensure that these are handled
Review Comment:
Thanks for catching this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3437217148
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
Review Comment:
nit: `loadCompletedTime` is a boxed `Long` but it's mandatory (we `checkNotNull` it below), so a primitive `long` would read a bit cleaner here - the boxed `Long` makes sense for the genuinely-nullable `loadCompletedIdleTimeoutMs` / `stateTtlMs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on PR #28329:
URL: https://github.com/apache/flink/pull/28329#issuecomment-4857381368
I've updated the PR and (hopefully) addressed or replied to all your comments.
Please take another look @rkhachatryan, @pnowojski, @gustavodemorais.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on PR #28329:
URL: https://github.com/apache/flink/pull/28329#issuecomment-4646543132
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on PR #28329:
URL: https://github.com/apache/flink/pull/28329#issuecomment-4968975948
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3552502961
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
Review Comment:
> What I meant is that the resulting value (returned by applyBufferedChangesIfReady) is hard to follow and easy to break. This is not a blocker though.
I agree, it's not straight-forward. The `bufferedAtWmState` is only set if the buffer holds change rows and in that case stores the current WM when the first change was buffered.
> if a late record comes in, and then we don't see a WM anymore; that record will sit stay buffered and will never be "used". Is that intended behavior?
I wouldn't say that this is intended, but it's the way that watermarks work. A window aggregation isn't flushed either when no WMs are received. We could of course add another time-out mechanism to flush the buffers after x seconds w/o watermarks, but that's a bunch of timers, state, and call-backs.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1078 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.EnumSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.StateBackendLoader;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffer is keyed by the
+ * build-side row-time attribute ({@code buildRowtimeIndex}) and applied in row-time order
+ * once the build watermark reaches it; changes sharing a row-time are applied in arrival
+ * order, which mirrors the upstream emit order (a retraction directly precedes its
+ * accumulation). Gating application on the row-time preserves atomic update visibility across
+ * {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD each is
+ * buffered in {@code probeBuffer} under a synthetic, increasing timestamp with one event-time
+ * timer registered per record. At the flip these timers fire (one record each, interruptibly)
+ * and join the buffered probes against the materialized build-side state in insertion order.
+ * During JOIN records are joined immediately, unless the key's flip drain has not finished
+ * yet, in which case the record is buffered behind the pending ones to preserve order.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ *
+ * <p>Streaming only: The operator joins data with processing-time semantics which can't be (easily)
+ * done in batch. Also, the operator's phase is held in union operator state, which is incompatible
+ * with finished operators. Joins against LATERAL SNAPSHOT functions should be translated to regular
+ * joins. The changes on the build-side would be consolidated into a final table and then be joined
+ * with the probe-side. All probe-side records are then joined against the same (and final) version
+ * of the build-side input.
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>,
+ Triggerable<RowData, VoidNamespace> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** Operator state names. */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** Keyed state names. */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String PROBE_BUFFER_SEQ_STATE_NAME = "probe-buffer-seq";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /**
+ * Single timer service for two kinds of per-key timers, distinguished by timer type (not by
+ * namespace): event-time timers drain a key's LOAD-buffered probes at the flip ({@link
+ * #onEventTime}), processing-time timers evict keyed state ({@link #onProcessingTime}). Both
+ * use {@link VoidNamespace} so no namespace is serialized per timer.
+ */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting
+ static final String NUM_PROBE_BUFFERED_METRIC_NAME = "numProbeSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting
+ static final String NUM_STATE_TTL_EVICTIONS_METRIC_NAME = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_BUILD_WM_METRIC_NAME = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_PROBE_WM_METRIC_NAME = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String CURRENT_PHASE_METRIC_NAME = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String MAX_JOIN_FAN_OUT_METRIC_NAME = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String AVG_JOIN_FAN_OUT_METRIC_NAME = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_PROBE_METRIC_NAME = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME =
+ "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<VoidNamespace> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ /**
+ * True if the keyed state backend iterates map entries in serialized-key order (RocksDB/ForSt),
+ * so the row-time-keyed {@link #buildChangeBuffer} is already scanned in event-time order.
+ */
+ private transient boolean sortedStateBackend;
+
+ // -------------------------- keyed state --------------------------
+
+ /**
+ * Build-side table as multi-set (row → reference count). Holds the committed snapshot, i.e. all
+ * build changes up to the last applied build watermark; probes join against this.
+ */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /**
+ * Build-side changes not yet visible in {@link #buildTableState}, keyed by their build row-time
+ * (multiple changes with the same row-time share a list value). Changes are applied to {@code
+ * buildTableState} once the build watermark reaches their row-time.
+ */
+ private transient MapState<Long, List<RowData>> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /**
+ * Buffer for probe-side records during LOAD (and for the rare JOIN-phase record that arrives
+ * while a key's buffer is still draining). Keyed by a synthetic, monotonically increasing
+ * timestamp so the per-record flip timers fire in insertion order. Each buffered row has one
+ * event-time timer registered on its key; the timer drains exactly that row.
+ */
+ private transient MapState<Long, RowData> probeBuffer;
+
+ /**
+ * Next sequence number for the current key, used to derive distinct, ordered synthetic
+ * timestamps for {@link #probeBuffer}. {@code null} iff the buffer is empty, so it doubles as a
+ * cheap "buffer non-empty" flag on the JOIN-phase fast path.
+ */
+ private transient ValueState<Long> probeBufferSeq;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<Phase> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ if (buildRowtimeIndex >= rightType.toRowType().getFieldCount()) {
+ throw new IllegalArgumentException("buildRowtimeIndex out of bounds for build row");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME,
+ new EnumSerializer<>(Phase.class)));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (Phase persistedPhase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (persistedPhase == Phase.LOAD) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ probeBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME,
+ Types.LONG,
+ Types.LIST(rightType)));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ PROBE_BUFFER_STATE_NAME, Types.LONG, leftType));
+ probeBufferSeq =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(
+ PROBE_BUFFER_SEQ_STATE_NAME, Types.LONG));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice. Both timer kinds use VoidNamespace; they are told apart by timer type
+ // (event-time -> flip drain, processing-time -> TTL eviction), not by namespace.
+ timerService =
+ getInternalTimerService(TIMER_SERVICE_NAME, VoidNamespaceSerializer.INSTANCE, this);
+
+ // RocksDB/ForSt iterate map entries in serialized-key order, so the row-time-keyed build
+ // change buffer is scanned in event-time order; unordered backends (heap) need a sort.
+ final String backendId = getKeyedStateBackend().getBackendTypeIdentifier();
+ sortedStateBackend =
+ StateBackendLoader.ROCKSDB_STATE_BACKEND_NAME.equals(backendId)
+ || StateBackendLoader.FORST_STATE_BACKEND_NAME.equals(backendId);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(NUM_STATE_TTL_EVICTIONS_METRIC_NAME);
+ numUnmatchedProbeRecords = metricGroup.counter(NUM_UNMATCHED_PROBE_METRIC_NAME);
+ numUnmatchedBuildRetractions =
+ metricGroup.counter(NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME);
+
+ // Best-effort in-memory tally; not restored, so it starts at 0 after a restore/rescale.
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(NUM_PROBE_BUFFERED_METRIC_NAME, probeBufferedGauge);
+ metricGroup.gauge(CURRENT_BUILD_WM_METRIC_NAME, buildWmGauge);
+ metricGroup.gauge(CURRENT_PROBE_WM_METRIC_NAME, probeWmGauge);
+ metricGroup.gauge(MAX_JOIN_FAN_OUT_METRIC_NAME, maxFanOutGauge);
+ metricGroup.gauge(AVG_JOIN_FAN_OUT_METRIC_NAME, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(CURRENT_PHASE_METRIC_NAME, phaseGauge);
+
+ // Pin the build-side input idle: the operator absorbs build-side watermarks and status.
+ combinedWatermark.updateStatus(1, true);
+
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ // Buffer during LOAD; in JOIN, buffer too if this key's flip drain has not finished yet
+ // (probeBufferSeq != null), so the new probe is joined in order after the buffered ones by
+ // its own timer instead of draining the whole buffer inline. Otherwise join immediately.
+ if (phase == Phase.LOAD || probeBufferSeq.value() != null) {
+ bufferProbe(probe);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ /**
+ * Buffers a probe row for the current key and registers a per-record event-time flip timer. The
+ * synthetic timestamp {@code (seq + 1) - Long.MAX_VALUE} is negative (so any non-negative
+ * watermark fires it) and increases with the sequence number (so timers drain in insertion
+ * order on every state backend).
+ */
+ private void bufferProbe(RowData probe) throws Exception {
+ long seq = probeBufferSeq.value() == null ? 0L : probeBufferSeq.value();
+ long ts = (seq + 1) - Long.MAX_VALUE;
+ probeBuffer.put(ts, probe);
+ probeBufferSeq.update(seq + 1);
+ timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, ts);
+ probeBufferedGauge.update(++probeBufferedCount);
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the change under its build row-time (grouping changes that share a row-time).
+ long rowtime = build.getLong(buildRowtimeIndex);
+ List<RowData> atRowtime = buildChangeBuffer.get(rowtime);
+ if (atRowtime == null) {
+ atRowtime = new ArrayList<>();
+ }
+ atRowtime.add(build);
+ buildChangeBuffer.put(rowtime, atRowtime);
+ // Tag the buffer with the current build watermark so it drains once the watermark passes.
+ // The tag may move backwards on restore (currentBuildSideWm resets to MIN_VALUE), which
+ // re-anchors a recovered buffer to this subtask's watermark.
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed; combined watermark is probe-driven.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // LOAD emits nothing downstream; track partial[0]'s idle bit for use after the flip.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only event-time (flip) timers are registered: each drains exactly one buffered probe for
+ // its key. All of a key's timers become due at the flip; interruptible timers fire them one
+ // at a time, yielding to checkpoints between firings.
+ long ts = timer.getTimestamp();
+ // Apply this key's due build-side changes before joining, so probes see the materialized
+ // build snapshot. Idempotent across the key's firings within one watermark sweep.
+ applyDueBufferedChanges();
+ RowData probe = probeBuffer.get(ts);
+ if (probe == null) {
+ // Already drained or evicted (e.g. by TTL) before this timer fired.
+ return;
+ }
+ joinProbeRow(probe);
+ probeBuffer.remove(ts);
+ probeBufferedCount = Math.max(0, probeBufferedCount - 1);
+ probeBufferedGauge.update(probeBufferedCount);
+ // This is the last buffered row iff it carries the highest sequence number; clearing the
+ // sequence here marks the buffer empty (cheap point lookup, no isEmpty() scan).
+ Long seq = probeBufferSeq.value();
+ if (seq != null && ts + Long.MAX_VALUE == seq) {
+ probeBufferSeq.clear();
+ }
+ }
+
+ /** Removes all probe-buffer state for the current key. */
+ private void clearProbeBuffer() {
+ probeBuffer.clear();
+ probeBufferSeq.clear();
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only processing-time (TTL) timers are registered. Semantics match Flink's standard
+ // StateTtlConfig.
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // Reschedule if still in LOAD, or in JOIN but within stateTtlMs of the flip (grace window).
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(VoidNamespace.INSTANCE, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ clearAllPerKeyState();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN. Runs in a NON-KEYED context (callers {@link #processWatermark2}
+ * and {@link #idleFlipTimer}), so it must not touch keyed state directly; the per-key probe
+ * drain happens in the per-record event-time timers fired below.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Anchor the TTL grace window at the flip so keys loaded before it aren't evicted early.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ cancelIdleFlipTimer();
+
+ // If we have seen a (non-negative) probe-side WM, forward it so the per-record flip timers
+ // fire (their synthetic timestamps are negative) and drain the buffered probes. Firing is
+ // interruptible and the watermark is forwarded after the buffered probes are joined.
+ // If we haven't seen a probe-side WM yet, draining is triggered by the next probe-side WM.
+ if (currentProbeSideWm >= 0) {
+ super.processWatermark1(new Watermark(currentProbeSideWm));
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm >= 0 ? currentProbeSideWm : "none");
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // The build-side wm advanced: apply the changes that are now due (their row-time has
+ // been reached by the build-side watermark) and keep any not-yet-due changes buffered.
+ applyDueBufferedChanges();
+ return bufferedAtWmState.value();
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark seen by this subtask (recovery/rescale,
+ // or idle-timeout flip). Force-apply everything now, since there is no watermark to
+ // gate on and the next build watermark may not arrive for a long time.
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Applies the buffered build-side changes that are due — i.e. whose row-time has been reached
+ * by the build-side watermark — in event-time order, leaving any not-yet-due changes untouched
+ * in state.
+ */
+ private void applyDueBufferedChanges() throws Exception {
Review Comment:
Good point!
I've refactored the different `applyBufferedChanges` methods to make the code easier to understand and remove some state accessed.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
+ buildTableState.put(change, currentCnt == null ? 1L : currentCnt + 1L);
+ } else {
+ // -D / -U
+ if (currentCnt == null || currentCnt <= 0L) {
+ numUnmatchedBuildRetractions.inc();
+ return;
+ }
Review Comment:
yes, that's possible. But there's also no guarantee that the corresponding inserts would ever show up so we might just buffer them forever.
Compared to user-cause out-of-order messages, the (IMO) more common scenario is that the job starts somewhere in the middle of a changelog and non-matching deletes can be ignored because the corresponding inserts will never be read.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1078 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.EnumSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.StateBackendLoader;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffer is keyed by the
+ * build-side row-time attribute ({@code buildRowtimeIndex}) and applied in row-time order
+ * once the build watermark reaches it; changes sharing a row-time are applied in arrival
+ * order, which mirrors the upstream emit order (a retraction directly precedes its
+ * accumulation). Gating application on the row-time preserves atomic update visibility across
+ * {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD each is
+ * buffered in {@code probeBuffer} under a synthetic, increasing timestamp with one event-time
+ * timer registered per record. At the flip these timers fire (one record each, interruptibly)
+ * and join the buffered probes against the materialized build-side state in insertion order.
+ * During JOIN records are joined immediately, unless the key's flip drain has not finished
+ * yet, in which case the record is buffered behind the pending ones to preserve order.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ *
+ * <p>Streaming only: The operator joins data with processing-time semantics which can't be (easily)
+ * done in batch. Also, the operator's phase is held in union operator state, which is incompatible
+ * with finished operators. Joins against LATERAL SNAPSHOT functions should be translated to regular
+ * joins. The changes on the build-side would be consolidated into a final table and then be joined
+ * with the probe-side. All probe-side records are then joined against the same (and final) version
+ * of the build-side input.
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>,
+ Triggerable<RowData, VoidNamespace> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** Operator state names. */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** Keyed state names. */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String PROBE_BUFFER_SEQ_STATE_NAME = "probe-buffer-seq";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /**
+ * Single timer service for two kinds of per-key timers, distinguished by timer type (not by
+ * namespace): event-time timers drain a key's LOAD-buffered probes at the flip ({@link
+ * #onEventTime}), processing-time timers evict keyed state ({@link #onProcessingTime}). Both
+ * use {@link VoidNamespace} so no namespace is serialized per timer.
+ */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting
+ static final String NUM_PROBE_BUFFERED_METRIC_NAME = "numProbeSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting
+ static final String NUM_STATE_TTL_EVICTIONS_METRIC_NAME = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_BUILD_WM_METRIC_NAME = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_PROBE_WM_METRIC_NAME = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String CURRENT_PHASE_METRIC_NAME = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String MAX_JOIN_FAN_OUT_METRIC_NAME = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String AVG_JOIN_FAN_OUT_METRIC_NAME = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_PROBE_METRIC_NAME = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME =
+ "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<VoidNamespace> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ /**
+ * True if the keyed state backend iterates map entries in serialized-key order (RocksDB/ForSt),
+ * so the row-time-keyed {@link #buildChangeBuffer} is already scanned in event-time order.
+ */
+ private transient boolean sortedStateBackend;
+
+ // -------------------------- keyed state --------------------------
+
+ /**
+ * Build-side table as multi-set (row → reference count). Holds the committed snapshot, i.e. all
+ * build changes up to the last applied build watermark; probes join against this.
+ */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /**
+ * Build-side changes not yet visible in {@link #buildTableState}, keyed by their build row-time
+ * (multiple changes with the same row-time share a list value). Changes are applied to {@code
+ * buildTableState} once the build watermark reaches their row-time.
+ */
+ private transient MapState<Long, List<RowData>> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /**
+ * Buffer for probe-side records during LOAD (and for the rare JOIN-phase record that arrives
+ * while a key's buffer is still draining). Keyed by a synthetic, monotonically increasing
+ * timestamp so the per-record flip timers fire in insertion order. Each buffered row has one
+ * event-time timer registered on its key; the timer drains exactly that row.
+ */
+ private transient MapState<Long, RowData> probeBuffer;
+
+ /**
+ * Next sequence number for the current key, used to derive distinct, ordered synthetic
+ * timestamps for {@link #probeBuffer}. {@code null} iff the buffer is empty, so it doubles as a
+ * cheap "buffer non-empty" flag on the JOIN-phase fast path.
+ */
+ private transient ValueState<Long> probeBufferSeq;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<Phase> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ if (buildRowtimeIndex >= rightType.toRowType().getFieldCount()) {
+ throw new IllegalArgumentException("buildRowtimeIndex out of bounds for build row");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME,
+ new EnumSerializer<>(Phase.class)));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (Phase persistedPhase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (persistedPhase == Phase.LOAD) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ probeBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME,
+ Types.LONG,
+ Types.LIST(rightType)));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ PROBE_BUFFER_STATE_NAME, Types.LONG, leftType));
+ probeBufferSeq =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(
+ PROBE_BUFFER_SEQ_STATE_NAME, Types.LONG));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice. Both timer kinds use VoidNamespace; they are told apart by timer type
+ // (event-time -> flip drain, processing-time -> TTL eviction), not by namespace.
+ timerService =
+ getInternalTimerService(TIMER_SERVICE_NAME, VoidNamespaceSerializer.INSTANCE, this);
+
+ // RocksDB/ForSt iterate map entries in serialized-key order, so the row-time-keyed build
+ // change buffer is scanned in event-time order; unordered backends (heap) need a sort.
+ final String backendId = getKeyedStateBackend().getBackendTypeIdentifier();
+ sortedStateBackend =
+ StateBackendLoader.ROCKSDB_STATE_BACKEND_NAME.equals(backendId)
+ || StateBackendLoader.FORST_STATE_BACKEND_NAME.equals(backendId);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(NUM_STATE_TTL_EVICTIONS_METRIC_NAME);
+ numUnmatchedProbeRecords = metricGroup.counter(NUM_UNMATCHED_PROBE_METRIC_NAME);
+ numUnmatchedBuildRetractions =
+ metricGroup.counter(NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME);
+
+ // Best-effort in-memory tally; not restored, so it starts at 0 after a restore/rescale.
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(NUM_PROBE_BUFFERED_METRIC_NAME, probeBufferedGauge);
+ metricGroup.gauge(CURRENT_BUILD_WM_METRIC_NAME, buildWmGauge);
+ metricGroup.gauge(CURRENT_PROBE_WM_METRIC_NAME, probeWmGauge);
+ metricGroup.gauge(MAX_JOIN_FAN_OUT_METRIC_NAME, maxFanOutGauge);
+ metricGroup.gauge(AVG_JOIN_FAN_OUT_METRIC_NAME, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(CURRENT_PHASE_METRIC_NAME, phaseGauge);
+
+ // Pin the build-side input idle: the operator absorbs build-side watermarks and status.
+ combinedWatermark.updateStatus(1, true);
+
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ // Buffer during LOAD; in JOIN, buffer too if this key's flip drain has not finished yet
+ // (probeBufferSeq != null), so the new probe is joined in order after the buffered ones by
+ // its own timer instead of draining the whole buffer inline. Otherwise join immediately.
+ if (phase == Phase.LOAD || probeBufferSeq.value() != null) {
+ bufferProbe(probe);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ /**
+ * Buffers a probe row for the current key and registers a per-record event-time flip timer. The
+ * synthetic timestamp {@code (seq + 1) - Long.MAX_VALUE} is negative (so any non-negative
+ * watermark fires it) and increases with the sequence number (so timers drain in insertion
+ * order on every state backend).
+ */
+ private void bufferProbe(RowData probe) throws Exception {
+ long seq = probeBufferSeq.value() == null ? 0L : probeBufferSeq.value();
+ long ts = (seq + 1) - Long.MAX_VALUE;
+ probeBuffer.put(ts, probe);
+ probeBufferSeq.update(seq + 1);
+ timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, ts);
+ probeBufferedGauge.update(++probeBufferedCount);
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the change under its build row-time (grouping changes that share a row-time).
+ long rowtime = build.getLong(buildRowtimeIndex);
+ List<RowData> atRowtime = buildChangeBuffer.get(rowtime);
+ if (atRowtime == null) {
+ atRowtime = new ArrayList<>();
+ }
+ atRowtime.add(build);
+ buildChangeBuffer.put(rowtime, atRowtime);
+ // Tag the buffer with the current build watermark so it drains once the watermark passes.
+ // The tag may move backwards on restore (currentBuildSideWm resets to MIN_VALUE), which
+ // re-anchors a recovered buffer to this subtask's watermark.
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed; combined watermark is probe-driven.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // LOAD emits nothing downstream; track partial[0]'s idle bit for use after the flip.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only event-time (flip) timers are registered: each drains exactly one buffered probe for
+ // its key. All of a key's timers become due at the flip; interruptible timers fire them one
+ // at a time, yielding to checkpoints between firings.
+ long ts = timer.getTimestamp();
+ // Apply this key's due build-side changes before joining, so probes see the materialized
+ // build snapshot. Idempotent across the key's firings within one watermark sweep.
+ applyDueBufferedChanges();
+ RowData probe = probeBuffer.get(ts);
+ if (probe == null) {
+ // Already drained or evicted (e.g. by TTL) before this timer fired.
+ return;
+ }
+ joinProbeRow(probe);
+ probeBuffer.remove(ts);
+ probeBufferedCount = Math.max(0, probeBufferedCount - 1);
+ probeBufferedGauge.update(probeBufferedCount);
+ // This is the last buffered row iff it carries the highest sequence number; clearing the
+ // sequence here marks the buffer empty (cheap point lookup, no isEmpty() scan).
+ Long seq = probeBufferSeq.value();
+ if (seq != null && ts + Long.MAX_VALUE == seq) {
Review Comment:
I'll add a check that `ts <= 0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3547439132
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
Review Comment:
Makes sense, thanks for the clarification.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1078 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.EnumSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.StateBackendLoader;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffer is keyed by the
+ * build-side row-time attribute ({@code buildRowtimeIndex}) and applied in row-time order
+ * once the build watermark reaches it; changes sharing a row-time are applied in arrival
+ * order, which mirrors the upstream emit order (a retraction directly precedes its
+ * accumulation). Gating application on the row-time preserves atomic update visibility across
+ * {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD each is
+ * buffered in {@code probeBuffer} under a synthetic, increasing timestamp with one event-time
+ * timer registered per record. At the flip these timers fire (one record each, interruptibly)
+ * and join the buffered probes against the materialized build-side state in insertion order.
+ * During JOIN records are joined immediately, unless the key's flip drain has not finished
+ * yet, in which case the record is buffered behind the pending ones to preserve order.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ *
+ * <p>Streaming only: The operator joins data with processing-time semantics which can't be (easily)
+ * done in batch. Also, the operator's phase is held in union operator state, which is incompatible
+ * with finished operators. Joins against LATERAL SNAPSHOT functions should be translated to regular
+ * joins. The changes on the build-side would be consolidated into a final table and then be joined
+ * with the probe-side. All probe-side records are then joined against the same (and final) version
+ * of the build-side input.
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>,
+ Triggerable<RowData, VoidNamespace> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** Operator state names. */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** Keyed state names. */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String PROBE_BUFFER_SEQ_STATE_NAME = "probe-buffer-seq";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /**
+ * Single timer service for two kinds of per-key timers, distinguished by timer type (not by
+ * namespace): event-time timers drain a key's LOAD-buffered probes at the flip ({@link
+ * #onEventTime}), processing-time timers evict keyed state ({@link #onProcessingTime}). Both
+ * use {@link VoidNamespace} so no namespace is serialized per timer.
+ */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting
+ static final String NUM_PROBE_BUFFERED_METRIC_NAME = "numProbeSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting
+ static final String NUM_STATE_TTL_EVICTIONS_METRIC_NAME = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_BUILD_WM_METRIC_NAME = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_PROBE_WM_METRIC_NAME = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String CURRENT_PHASE_METRIC_NAME = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String MAX_JOIN_FAN_OUT_METRIC_NAME = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String AVG_JOIN_FAN_OUT_METRIC_NAME = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_PROBE_METRIC_NAME = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME =
+ "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<VoidNamespace> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ /**
+ * True if the keyed state backend iterates map entries in serialized-key order (RocksDB/ForSt),
+ * so the row-time-keyed {@link #buildChangeBuffer} is already scanned in event-time order.
+ */
+ private transient boolean sortedStateBackend;
+
+ // -------------------------- keyed state --------------------------
+
+ /**
+ * Build-side table as multi-set (row → reference count). Holds the committed snapshot, i.e. all
+ * build changes up to the last applied build watermark; probes join against this.
+ */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /**
+ * Build-side changes not yet visible in {@link #buildTableState}, keyed by their build row-time
+ * (multiple changes with the same row-time share a list value). Changes are applied to {@code
+ * buildTableState} once the build watermark reaches their row-time.
+ */
+ private transient MapState<Long, List<RowData>> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /**
+ * Buffer for probe-side records during LOAD (and for the rare JOIN-phase record that arrives
+ * while a key's buffer is still draining). Keyed by a synthetic, monotonically increasing
+ * timestamp so the per-record flip timers fire in insertion order. Each buffered row has one
+ * event-time timer registered on its key; the timer drains exactly that row.
+ */
+ private transient MapState<Long, RowData> probeBuffer;
+
+ /**
+ * Next sequence number for the current key, used to derive distinct, ordered synthetic
+ * timestamps for {@link #probeBuffer}. {@code null} iff the buffer is empty, so it doubles as a
+ * cheap "buffer non-empty" flag on the JOIN-phase fast path.
+ */
+ private transient ValueState<Long> probeBufferSeq;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<Phase> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ if (buildRowtimeIndex >= rightType.toRowType().getFieldCount()) {
+ throw new IllegalArgumentException("buildRowtimeIndex out of bounds for build row");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME,
+ new EnumSerializer<>(Phase.class)));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (Phase persistedPhase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (persistedPhase == Phase.LOAD) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ probeBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME,
+ Types.LONG,
+ Types.LIST(rightType)));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ PROBE_BUFFER_STATE_NAME, Types.LONG, leftType));
+ probeBufferSeq =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(
+ PROBE_BUFFER_SEQ_STATE_NAME, Types.LONG));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice. Both timer kinds use VoidNamespace; they are told apart by timer type
+ // (event-time -> flip drain, processing-time -> TTL eviction), not by namespace.
+ timerService =
+ getInternalTimerService(TIMER_SERVICE_NAME, VoidNamespaceSerializer.INSTANCE, this);
+
+ // RocksDB/ForSt iterate map entries in serialized-key order, so the row-time-keyed build
+ // change buffer is scanned in event-time order; unordered backends (heap) need a sort.
+ final String backendId = getKeyedStateBackend().getBackendTypeIdentifier();
+ sortedStateBackend =
+ StateBackendLoader.ROCKSDB_STATE_BACKEND_NAME.equals(backendId)
+ || StateBackendLoader.FORST_STATE_BACKEND_NAME.equals(backendId);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(NUM_STATE_TTL_EVICTIONS_METRIC_NAME);
+ numUnmatchedProbeRecords = metricGroup.counter(NUM_UNMATCHED_PROBE_METRIC_NAME);
+ numUnmatchedBuildRetractions =
+ metricGroup.counter(NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME);
+
+ // Best-effort in-memory tally; not restored, so it starts at 0 after a restore/rescale.
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(NUM_PROBE_BUFFERED_METRIC_NAME, probeBufferedGauge);
+ metricGroup.gauge(CURRENT_BUILD_WM_METRIC_NAME, buildWmGauge);
+ metricGroup.gauge(CURRENT_PROBE_WM_METRIC_NAME, probeWmGauge);
+ metricGroup.gauge(MAX_JOIN_FAN_OUT_METRIC_NAME, maxFanOutGauge);
+ metricGroup.gauge(AVG_JOIN_FAN_OUT_METRIC_NAME, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(CURRENT_PHASE_METRIC_NAME, phaseGauge);
+
+ // Pin the build-side input idle: the operator absorbs build-side watermarks and status.
+ combinedWatermark.updateStatus(1, true);
+
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ // Buffer during LOAD; in JOIN, buffer too if this key's flip drain has not finished yet
+ // (probeBufferSeq != null), so the new probe is joined in order after the buffered ones by
+ // its own timer instead of draining the whole buffer inline. Otherwise join immediately.
+ if (phase == Phase.LOAD || probeBufferSeq.value() != null) {
+ bufferProbe(probe);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ /**
+ * Buffers a probe row for the current key and registers a per-record event-time flip timer. The
+ * synthetic timestamp {@code (seq + 1) - Long.MAX_VALUE} is negative (so any non-negative
+ * watermark fires it) and increases with the sequence number (so timers drain in insertion
+ * order on every state backend).
+ */
+ private void bufferProbe(RowData probe) throws Exception {
+ long seq = probeBufferSeq.value() == null ? 0L : probeBufferSeq.value();
+ long ts = (seq + 1) - Long.MAX_VALUE;
+ probeBuffer.put(ts, probe);
+ probeBufferSeq.update(seq + 1);
+ timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, ts);
+ probeBufferedGauge.update(++probeBufferedCount);
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the change under its build row-time (grouping changes that share a row-time).
+ long rowtime = build.getLong(buildRowtimeIndex);
+ List<RowData> atRowtime = buildChangeBuffer.get(rowtime);
+ if (atRowtime == null) {
+ atRowtime = new ArrayList<>();
+ }
+ atRowtime.add(build);
+ buildChangeBuffer.put(rowtime, atRowtime);
+ // Tag the buffer with the current build watermark so it drains once the watermark passes.
+ // The tag may move backwards on restore (currentBuildSideWm resets to MIN_VALUE), which
+ // re-anchors a recovered buffer to this subtask's watermark.
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed; combined watermark is probe-driven.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // LOAD emits nothing downstream; track partial[0]'s idle bit for use after the flip.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only event-time (flip) timers are registered: each drains exactly one buffered probe for
+ // its key. All of a key's timers become due at the flip; interruptible timers fire them one
+ // at a time, yielding to checkpoints between firings.
+ long ts = timer.getTimestamp();
+ // Apply this key's due build-side changes before joining, so probes see the materialized
+ // build snapshot. Idempotent across the key's firings within one watermark sweep.
+ applyDueBufferedChanges();
+ RowData probe = probeBuffer.get(ts);
+ if (probe == null) {
+ // Already drained or evicted (e.g. by TTL) before this timer fired.
+ return;
+ }
+ joinProbeRow(probe);
+ probeBuffer.remove(ts);
+ probeBufferedCount = Math.max(0, probeBufferedCount - 1);
+ probeBufferedGauge.update(probeBufferedCount);
+ // This is the last buffered row iff it carries the highest sequence number; clearing the
+ // sequence here marks the buffer empty (cheap point lookup, no isEmpty() scan).
+ Long seq = probeBufferSeq.value();
+ if (seq != null && ts + Long.MAX_VALUE == seq) {
Review Comment:
Should we sanity-check that there's no overflow in `ts + Long.MAX_VALUE `?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1078 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.EnumSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.StateBackendLoader;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffer is keyed by the
+ * build-side row-time attribute ({@code buildRowtimeIndex}) and applied in row-time order
+ * once the build watermark reaches it; changes sharing a row-time are applied in arrival
+ * order, which mirrors the upstream emit order (a retraction directly precedes its
+ * accumulation). Gating application on the row-time preserves atomic update visibility across
+ * {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD each is
+ * buffered in {@code probeBuffer} under a synthetic, increasing timestamp with one event-time
+ * timer registered per record. At the flip these timers fire (one record each, interruptibly)
+ * and join the buffered probes against the materialized build-side state in insertion order.
+ * During JOIN records are joined immediately, unless the key's flip drain has not finished
+ * yet, in which case the record is buffered behind the pending ones to preserve order.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ *
+ * <p>Streaming only: The operator joins data with processing-time semantics which can't be (easily)
+ * done in batch. Also, the operator's phase is held in union operator state, which is incompatible
+ * with finished operators. Joins against LATERAL SNAPSHOT functions should be translated to regular
+ * joins. The changes on the build-side would be consolidated into a final table and then be joined
+ * with the probe-side. All probe-side records are then joined against the same (and final) version
+ * of the build-side input.
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>,
+ Triggerable<RowData, VoidNamespace> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** Operator state names. */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** Keyed state names. */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String PROBE_BUFFER_SEQ_STATE_NAME = "probe-buffer-seq";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /**
+ * Single timer service for two kinds of per-key timers, distinguished by timer type (not by
+ * namespace): event-time timers drain a key's LOAD-buffered probes at the flip ({@link
+ * #onEventTime}), processing-time timers evict keyed state ({@link #onProcessingTime}). Both
+ * use {@link VoidNamespace} so no namespace is serialized per timer.
+ */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting
+ static final String NUM_PROBE_BUFFERED_METRIC_NAME = "numProbeSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting
+ static final String NUM_STATE_TTL_EVICTIONS_METRIC_NAME = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_BUILD_WM_METRIC_NAME = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_PROBE_WM_METRIC_NAME = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String CURRENT_PHASE_METRIC_NAME = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String MAX_JOIN_FAN_OUT_METRIC_NAME = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String AVG_JOIN_FAN_OUT_METRIC_NAME = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_PROBE_METRIC_NAME = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME =
+ "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<VoidNamespace> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ /**
+ * True if the keyed state backend iterates map entries in serialized-key order (RocksDB/ForSt),
+ * so the row-time-keyed {@link #buildChangeBuffer} is already scanned in event-time order.
+ */
+ private transient boolean sortedStateBackend;
+
+ // -------------------------- keyed state --------------------------
+
+ /**
+ * Build-side table as multi-set (row → reference count). Holds the committed snapshot, i.e. all
+ * build changes up to the last applied build watermark; probes join against this.
+ */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /**
+ * Build-side changes not yet visible in {@link #buildTableState}, keyed by their build row-time
+ * (multiple changes with the same row-time share a list value). Changes are applied to {@code
+ * buildTableState} once the build watermark reaches their row-time.
+ */
+ private transient MapState<Long, List<RowData>> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /**
+ * Buffer for probe-side records during LOAD (and for the rare JOIN-phase record that arrives
+ * while a key's buffer is still draining). Keyed by a synthetic, monotonically increasing
+ * timestamp so the per-record flip timers fire in insertion order. Each buffered row has one
+ * event-time timer registered on its key; the timer drains exactly that row.
+ */
+ private transient MapState<Long, RowData> probeBuffer;
+
+ /**
+ * Next sequence number for the current key, used to derive distinct, ordered synthetic
+ * timestamps for {@link #probeBuffer}. {@code null} iff the buffer is empty, so it doubles as a
+ * cheap "buffer non-empty" flag on the JOIN-phase fast path.
+ */
+ private transient ValueState<Long> probeBufferSeq;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<Phase> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ if (buildRowtimeIndex >= rightType.toRowType().getFieldCount()) {
+ throw new IllegalArgumentException("buildRowtimeIndex out of bounds for build row");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME,
+ new EnumSerializer<>(Phase.class)));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (Phase persistedPhase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (persistedPhase == Phase.LOAD) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ probeBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME,
+ Types.LONG,
+ Types.LIST(rightType)));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ PROBE_BUFFER_STATE_NAME, Types.LONG, leftType));
+ probeBufferSeq =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(
+ PROBE_BUFFER_SEQ_STATE_NAME, Types.LONG));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice. Both timer kinds use VoidNamespace; they are told apart by timer type
+ // (event-time -> flip drain, processing-time -> TTL eviction), not by namespace.
+ timerService =
+ getInternalTimerService(TIMER_SERVICE_NAME, VoidNamespaceSerializer.INSTANCE, this);
+
+ // RocksDB/ForSt iterate map entries in serialized-key order, so the row-time-keyed build
+ // change buffer is scanned in event-time order; unordered backends (heap) need a sort.
+ final String backendId = getKeyedStateBackend().getBackendTypeIdentifier();
+ sortedStateBackend =
+ StateBackendLoader.ROCKSDB_STATE_BACKEND_NAME.equals(backendId)
+ || StateBackendLoader.FORST_STATE_BACKEND_NAME.equals(backendId);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(NUM_STATE_TTL_EVICTIONS_METRIC_NAME);
+ numUnmatchedProbeRecords = metricGroup.counter(NUM_UNMATCHED_PROBE_METRIC_NAME);
+ numUnmatchedBuildRetractions =
+ metricGroup.counter(NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME);
+
+ // Best-effort in-memory tally; not restored, so it starts at 0 after a restore/rescale.
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(NUM_PROBE_BUFFERED_METRIC_NAME, probeBufferedGauge);
+ metricGroup.gauge(CURRENT_BUILD_WM_METRIC_NAME, buildWmGauge);
+ metricGroup.gauge(CURRENT_PROBE_WM_METRIC_NAME, probeWmGauge);
+ metricGroup.gauge(MAX_JOIN_FAN_OUT_METRIC_NAME, maxFanOutGauge);
+ metricGroup.gauge(AVG_JOIN_FAN_OUT_METRIC_NAME, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(CURRENT_PHASE_METRIC_NAME, phaseGauge);
+
+ // Pin the build-side input idle: the operator absorbs build-side watermarks and status.
+ combinedWatermark.updateStatus(1, true);
+
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ // Buffer during LOAD; in JOIN, buffer too if this key's flip drain has not finished yet
+ // (probeBufferSeq != null), so the new probe is joined in order after the buffered ones by
+ // its own timer instead of draining the whole buffer inline. Otherwise join immediately.
+ if (phase == Phase.LOAD || probeBufferSeq.value() != null) {
+ bufferProbe(probe);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ /**
+ * Buffers a probe row for the current key and registers a per-record event-time flip timer. The
+ * synthetic timestamp {@code (seq + 1) - Long.MAX_VALUE} is negative (so any non-negative
+ * watermark fires it) and increases with the sequence number (so timers drain in insertion
+ * order on every state backend).
+ */
+ private void bufferProbe(RowData probe) throws Exception {
+ long seq = probeBufferSeq.value() == null ? 0L : probeBufferSeq.value();
+ long ts = (seq + 1) - Long.MAX_VALUE;
+ probeBuffer.put(ts, probe);
+ probeBufferSeq.update(seq + 1);
+ timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, ts);
+ probeBufferedGauge.update(++probeBufferedCount);
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the change under its build row-time (grouping changes that share a row-time).
+ long rowtime = build.getLong(buildRowtimeIndex);
+ List<RowData> atRowtime = buildChangeBuffer.get(rowtime);
+ if (atRowtime == null) {
+ atRowtime = new ArrayList<>();
+ }
+ atRowtime.add(build);
+ buildChangeBuffer.put(rowtime, atRowtime);
+ // Tag the buffer with the current build watermark so it drains once the watermark passes.
+ // The tag may move backwards on restore (currentBuildSideWm resets to MIN_VALUE), which
+ // re-anchors a recovered buffer to this subtask's watermark.
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed; combined watermark is probe-driven.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // LOAD emits nothing downstream; track partial[0]'s idle bit for use after the flip.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only event-time (flip) timers are registered: each drains exactly one buffered probe for
+ // its key. All of a key's timers become due at the flip; interruptible timers fire them one
+ // at a time, yielding to checkpoints between firings.
+ long ts = timer.getTimestamp();
+ // Apply this key's due build-side changes before joining, so probes see the materialized
+ // build snapshot. Idempotent across the key's firings within one watermark sweep.
+ applyDueBufferedChanges();
+ RowData probe = probeBuffer.get(ts);
+ if (probe == null) {
+ // Already drained or evicted (e.g. by TTL) before this timer fired.
+ return;
+ }
+ joinProbeRow(probe);
+ probeBuffer.remove(ts);
+ probeBufferedCount = Math.max(0, probeBufferedCount - 1);
+ probeBufferedGauge.update(probeBufferedCount);
+ // This is the last buffered row iff it carries the highest sequence number; clearing the
+ // sequence here marks the buffer empty (cheap point lookup, no isEmpty() scan).
+ Long seq = probeBufferSeq.value();
+ if (seq != null && ts + Long.MAX_VALUE == seq) {
+ probeBufferSeq.clear();
+ }
+ }
+
+ /** Removes all probe-buffer state for the current key. */
+ private void clearProbeBuffer() {
+ probeBuffer.clear();
+ probeBufferSeq.clear();
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only processing-time (TTL) timers are registered. Semantics match Flink's standard
+ // StateTtlConfig.
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // Reschedule if still in LOAD, or in JOIN but within stateTtlMs of the flip (grace window).
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(VoidNamespace.INSTANCE, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ clearAllPerKeyState();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN. Runs in a NON-KEYED context (callers {@link #processWatermark2}
+ * and {@link #idleFlipTimer}), so it must not touch keyed state directly; the per-key probe
+ * drain happens in the per-record event-time timers fired below.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Anchor the TTL grace window at the flip so keys loaded before it aren't evicted early.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ cancelIdleFlipTimer();
+
+ // If we have seen a (non-negative) probe-side WM, forward it so the per-record flip timers
+ // fire (their synthetic timestamps are negative) and drain the buffered probes. Firing is
+ // interruptible and the watermark is forwarded after the buffered probes are joined.
+ // If we haven't seen a probe-side WM yet, draining is triggered by the next probe-side WM.
+ if (currentProbeSideWm >= 0) {
+ super.processWatermark1(new Watermark(currentProbeSideWm));
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm >= 0 ? currentProbeSideWm : "none");
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // The build-side wm advanced: apply the changes that are now due (their row-time has
+ // been reached by the build-side watermark) and keep any not-yet-due changes buffered.
+ applyDueBufferedChanges();
+ return bufferedAtWmState.value();
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark seen by this subtask (recovery/rescale,
+ // or idle-timeout flip). Force-apply everything now, since there is no watermark to
+ // gate on and the next build watermark may not arrive for a long time.
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Applies the buffered build-side changes that are due — i.e. whose row-time has been reached
+ * by the build-side watermark — in event-time order, leaving any not-yet-due changes untouched
+ * in state.
+ */
+ private void applyDueBufferedChanges() throws Exception {
Review Comment:
I think we could dedup much of the code in `applyBufferedChanges` and in this method.
For example, extract `apply(long upToWm)` and pass upToWm=Long.MAX_VALUE from `applyBufferedChanges`.
That could also help with eliminating state accesses.
WDYT?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
Review Comment:
What I meant is that the resulting value (returned by `applyBufferedChangesIfReady`) is hard to follow and easy to break. This is not a blocker though.
Separate question: if a late record comes in, and then we don't see a WM anymore; that record will sit stay buffered and will never be "used".
Is that intended behavior?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1078 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.EnumSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.StateBackendLoader;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffer is keyed by the
+ * build-side row-time attribute ({@code buildRowtimeIndex}) and applied in row-time order
+ * once the build watermark reaches it; changes sharing a row-time are applied in arrival
+ * order, which mirrors the upstream emit order (a retraction directly precedes its
+ * accumulation). Gating application on the row-time preserves atomic update visibility across
+ * {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD each is
+ * buffered in {@code probeBuffer} under a synthetic, increasing timestamp with one event-time
+ * timer registered per record. At the flip these timers fire (one record each, interruptibly)
+ * and join the buffered probes against the materialized build-side state in insertion order.
+ * During JOIN records are joined immediately, unless the key's flip drain has not finished
+ * yet, in which case the record is buffered behind the pending ones to preserve order.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ *
+ * <p>Streaming only: The operator joins data with processing-time semantics which can't be (easily)
+ * done in batch. Also, the operator's phase is held in union operator state, which is incompatible
+ * with finished operators. Joins against LATERAL SNAPSHOT functions should be translated to regular
+ * joins. The changes on the build-side would be consolidated into a final table and then be joined
+ * with the probe-side. All probe-side records are then joined against the same (and final) version
+ * of the build-side input.
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>,
+ Triggerable<RowData, VoidNamespace> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** Operator state names. */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** Keyed state names. */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String PROBE_BUFFER_SEQ_STATE_NAME = "probe-buffer-seq";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /**
+ * Single timer service for two kinds of per-key timers, distinguished by timer type (not by
+ * namespace): event-time timers drain a key's LOAD-buffered probes at the flip ({@link
+ * #onEventTime}), processing-time timers evict keyed state ({@link #onProcessingTime}). Both
+ * use {@link VoidNamespace} so no namespace is serialized per timer.
+ */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting
+ static final String NUM_PROBE_BUFFERED_METRIC_NAME = "numProbeSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting
+ static final String NUM_STATE_TTL_EVICTIONS_METRIC_NAME = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_BUILD_WM_METRIC_NAME = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_PROBE_WM_METRIC_NAME = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String CURRENT_PHASE_METRIC_NAME = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String MAX_JOIN_FAN_OUT_METRIC_NAME = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String AVG_JOIN_FAN_OUT_METRIC_NAME = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_PROBE_METRIC_NAME = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME =
+ "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<VoidNamespace> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ /**
+ * True if the keyed state backend iterates map entries in serialized-key order (RocksDB/ForSt),
+ * so the row-time-keyed {@link #buildChangeBuffer} is already scanned in event-time order.
+ */
+ private transient boolean sortedStateBackend;
+
+ // -------------------------- keyed state --------------------------
+
+ /**
+ * Build-side table as multi-set (row → reference count). Holds the committed snapshot, i.e. all
+ * build changes up to the last applied build watermark; probes join against this.
+ */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /**
+ * Build-side changes not yet visible in {@link #buildTableState}, keyed by their build row-time
+ * (multiple changes with the same row-time share a list value). Changes are applied to {@code
+ * buildTableState} once the build watermark reaches their row-time.
+ */
+ private transient MapState<Long, List<RowData>> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /**
+ * Buffer for probe-side records during LOAD (and for the rare JOIN-phase record that arrives
+ * while a key's buffer is still draining). Keyed by a synthetic, monotonically increasing
+ * timestamp so the per-record flip timers fire in insertion order. Each buffered row has one
+ * event-time timer registered on its key; the timer drains exactly that row.
+ */
+ private transient MapState<Long, RowData> probeBuffer;
+
+ /**
+ * Next sequence number for the current key, used to derive distinct, ordered synthetic
+ * timestamps for {@link #probeBuffer}. {@code null} iff the buffer is empty, so it doubles as a
+ * cheap "buffer non-empty" flag on the JOIN-phase fast path.
+ */
+ private transient ValueState<Long> probeBufferSeq;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<Phase> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ if (buildRowtimeIndex >= rightType.toRowType().getFieldCount()) {
+ throw new IllegalArgumentException("buildRowtimeIndex out of bounds for build row");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME,
+ new EnumSerializer<>(Phase.class)));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (Phase persistedPhase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (persistedPhase == Phase.LOAD) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ probeBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME,
+ Types.LONG,
+ Types.LIST(rightType)));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ PROBE_BUFFER_STATE_NAME, Types.LONG, leftType));
+ probeBufferSeq =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(
+ PROBE_BUFFER_SEQ_STATE_NAME, Types.LONG));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice. Both timer kinds use VoidNamespace; they are told apart by timer type
+ // (event-time -> flip drain, processing-time -> TTL eviction), not by namespace.
+ timerService =
+ getInternalTimerService(TIMER_SERVICE_NAME, VoidNamespaceSerializer.INSTANCE, this);
+
+ // RocksDB/ForSt iterate map entries in serialized-key order, so the row-time-keyed build
+ // change buffer is scanned in event-time order; unordered backends (heap) need a sort.
+ final String backendId = getKeyedStateBackend().getBackendTypeIdentifier();
+ sortedStateBackend =
+ StateBackendLoader.ROCKSDB_STATE_BACKEND_NAME.equals(backendId)
+ || StateBackendLoader.FORST_STATE_BACKEND_NAME.equals(backendId);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(NUM_STATE_TTL_EVICTIONS_METRIC_NAME);
+ numUnmatchedProbeRecords = metricGroup.counter(NUM_UNMATCHED_PROBE_METRIC_NAME);
+ numUnmatchedBuildRetractions =
+ metricGroup.counter(NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME);
+
+ // Best-effort in-memory tally; not restored, so it starts at 0 after a restore/rescale.
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(NUM_PROBE_BUFFERED_METRIC_NAME, probeBufferedGauge);
+ metricGroup.gauge(CURRENT_BUILD_WM_METRIC_NAME, buildWmGauge);
+ metricGroup.gauge(CURRENT_PROBE_WM_METRIC_NAME, probeWmGauge);
+ metricGroup.gauge(MAX_JOIN_FAN_OUT_METRIC_NAME, maxFanOutGauge);
+ metricGroup.gauge(AVG_JOIN_FAN_OUT_METRIC_NAME, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(CURRENT_PHASE_METRIC_NAME, phaseGauge);
+
+ // Pin the build-side input idle: the operator absorbs build-side watermarks and status.
+ combinedWatermark.updateStatus(1, true);
+
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ // Buffer during LOAD; in JOIN, buffer too if this key's flip drain has not finished yet
+ // (probeBufferSeq != null), so the new probe is joined in order after the buffered ones by
+ // its own timer instead of draining the whole buffer inline. Otherwise join immediately.
+ if (phase == Phase.LOAD || probeBufferSeq.value() != null) {
+ bufferProbe(probe);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ /**
+ * Buffers a probe row for the current key and registers a per-record event-time flip timer. The
+ * synthetic timestamp {@code (seq + 1) - Long.MAX_VALUE} is negative (so any non-negative
+ * watermark fires it) and increases with the sequence number (so timers drain in insertion
+ * order on every state backend).
+ */
+ private void bufferProbe(RowData probe) throws Exception {
+ long seq = probeBufferSeq.value() == null ? 0L : probeBufferSeq.value();
+ long ts = (seq + 1) - Long.MAX_VALUE;
+ probeBuffer.put(ts, probe);
+ probeBufferSeq.update(seq + 1);
+ timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, ts);
+ probeBufferedGauge.update(++probeBufferedCount);
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the change under its build row-time (grouping changes that share a row-time).
+ long rowtime = build.getLong(buildRowtimeIndex);
+ List<RowData> atRowtime = buildChangeBuffer.get(rowtime);
+ if (atRowtime == null) {
+ atRowtime = new ArrayList<>();
+ }
+ atRowtime.add(build);
+ buildChangeBuffer.put(rowtime, atRowtime);
+ // Tag the buffer with the current build watermark so it drains once the watermark passes.
+ // The tag may move backwards on restore (currentBuildSideWm resets to MIN_VALUE), which
+ // re-anchors a recovered buffer to this subtask's watermark.
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed; combined watermark is probe-driven.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // LOAD emits nothing downstream; track partial[0]'s idle bit for use after the flip.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only event-time (flip) timers are registered: each drains exactly one buffered probe for
+ // its key. All of a key's timers become due at the flip; interruptible timers fire them one
+ // at a time, yielding to checkpoints between firings.
+ long ts = timer.getTimestamp();
+ // Apply this key's due build-side changes before joining, so probes see the materialized
+ // build snapshot. Idempotent across the key's firings within one watermark sweep.
+ applyDueBufferedChanges();
+ RowData probe = probeBuffer.get(ts);
+ if (probe == null) {
+ // Already drained or evicted (e.g. by TTL) before this timer fired.
+ return;
+ }
+ joinProbeRow(probe);
+ probeBuffer.remove(ts);
+ probeBufferedCount = Math.max(0, probeBufferedCount - 1);
+ probeBufferedGauge.update(probeBufferedCount);
+ // This is the last buffered row iff it carries the highest sequence number; clearing the
+ // sequence here marks the buffer empty (cheap point lookup, no isEmpty() scan).
+ Long seq = probeBufferSeq.value();
+ if (seq != null && ts + Long.MAX_VALUE == seq) {
+ probeBufferSeq.clear();
+ }
+ }
+
+ /** Removes all probe-buffer state for the current key. */
+ private void clearProbeBuffer() {
+ probeBuffer.clear();
+ probeBufferSeq.clear();
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only processing-time (TTL) timers are registered. Semantics match Flink's standard
+ // StateTtlConfig.
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // Reschedule if still in LOAD, or in JOIN but within stateTtlMs of the flip (grace window).
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(VoidNamespace.INSTANCE, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ clearAllPerKeyState();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN. Runs in a NON-KEYED context (callers {@link #processWatermark2}
+ * and {@link #idleFlipTimer}), so it must not touch keyed state directly; the per-key probe
+ * drain happens in the per-record event-time timers fired below.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Anchor the TTL grace window at the flip so keys loaded before it aren't evicted early.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ cancelIdleFlipTimer();
+
+ // If we have seen a (non-negative) probe-side WM, forward it so the per-record flip timers
+ // fire (their synthetic timestamps are negative) and drain the buffered probes. Firing is
+ // interruptible and the watermark is forwarded after the buffered probes are joined.
+ // If we haven't seen a probe-side WM yet, draining is triggered by the next probe-side WM.
+ if (currentProbeSideWm >= 0) {
+ super.processWatermark1(new Watermark(currentProbeSideWm));
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm >= 0 ? currentProbeSideWm : "none");
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
Review Comment:
NIT: rename to something like `applyBufferedChangesUntilWm`?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
+ buildTableState.put(change, currentCnt == null ? 1L : currentCnt + 1L);
+ } else {
+ // -D / -U
+ if (currentCnt == null || currentCnt <= 0L) {
+ numUnmatchedBuildRetractions.inc();
+ return;
+ }
Review Comment:
The former; I'm concerned that we're dropping them.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1078 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.EnumSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.StateBackendLoader;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffer is keyed by the
+ * build-side row-time attribute ({@code buildRowtimeIndex}) and applied in row-time order
+ * once the build watermark reaches it; changes sharing a row-time are applied in arrival
+ * order, which mirrors the upstream emit order (a retraction directly precedes its
+ * accumulation). Gating application on the row-time preserves atomic update visibility across
+ * {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD each is
+ * buffered in {@code probeBuffer} under a synthetic, increasing timestamp with one event-time
+ * timer registered per record. At the flip these timers fire (one record each, interruptibly)
+ * and join the buffered probes against the materialized build-side state in insertion order.
+ * During JOIN records are joined immediately, unless the key's flip drain has not finished
+ * yet, in which case the record is buffered behind the pending ones to preserve order.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ *
+ * <p>Streaming only: The operator joins data with processing-time semantics which can't be (easily)
+ * done in batch. Also, the operator's phase is held in union operator state, which is incompatible
+ * with finished operators. Joins against LATERAL SNAPSHOT functions should be translated to regular
+ * joins. The changes on the build-side would be consolidated into a final table and then be joined
+ * with the probe-side. All probe-side records are then joined against the same (and final) version
+ * of the build-side input.
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>,
+ Triggerable<RowData, VoidNamespace> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** Operator state names. */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** Keyed state names. */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String PROBE_BUFFER_SEQ_STATE_NAME = "probe-buffer-seq";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /**
+ * Single timer service for two kinds of per-key timers, distinguished by timer type (not by
+ * namespace): event-time timers drain a key's LOAD-buffered probes at the flip ({@link
+ * #onEventTime}), processing-time timers evict keyed state ({@link #onProcessingTime}). Both
+ * use {@link VoidNamespace} so no namespace is serialized per timer.
+ */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting
+ static final String NUM_PROBE_BUFFERED_METRIC_NAME = "numProbeSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting
+ static final String NUM_STATE_TTL_EVICTIONS_METRIC_NAME = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_BUILD_WM_METRIC_NAME = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting
+ static final String CURRENT_PROBE_WM_METRIC_NAME = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String CURRENT_PHASE_METRIC_NAME = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String MAX_JOIN_FAN_OUT_METRIC_NAME = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String AVG_JOIN_FAN_OUT_METRIC_NAME = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_PROBE_METRIC_NAME = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME =
+ "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<VoidNamespace> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ /**
+ * True if the keyed state backend iterates map entries in serialized-key order (RocksDB/ForSt),
+ * so the row-time-keyed {@link #buildChangeBuffer} is already scanned in event-time order.
+ */
+ private transient boolean sortedStateBackend;
+
+ // -------------------------- keyed state --------------------------
+
+ /**
+ * Build-side table as multi-set (row → reference count). Holds the committed snapshot, i.e. all
+ * build changes up to the last applied build watermark; probes join against this.
+ */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /**
+ * Build-side changes not yet visible in {@link #buildTableState}, keyed by their build row-time
+ * (multiple changes with the same row-time share a list value). Changes are applied to {@code
+ * buildTableState} once the build watermark reaches their row-time.
+ */
+ private transient MapState<Long, List<RowData>> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /**
+ * Buffer for probe-side records during LOAD (and for the rare JOIN-phase record that arrives
+ * while a key's buffer is still draining). Keyed by a synthetic, monotonically increasing
+ * timestamp so the per-record flip timers fire in insertion order. Each buffered row has one
+ * event-time timer registered on its key; the timer drains exactly that row.
+ */
+ private transient MapState<Long, RowData> probeBuffer;
+
+ /**
+ * Next sequence number for the current key, used to derive distinct, ordered synthetic
+ * timestamps for {@link #probeBuffer}. {@code null} iff the buffer is empty, so it doubles as a
+ * cheap "buffer non-empty" flag on the JOIN-phase fast path.
+ */
+ private transient ValueState<Long> probeBufferSeq;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<Phase> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ if (buildRowtimeIndex >= rightType.toRowType().getFieldCount()) {
+ throw new IllegalArgumentException("buildRowtimeIndex out of bounds for build row");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME,
+ new EnumSerializer<>(Phase.class)));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (Phase persistedPhase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (persistedPhase == Phase.LOAD) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ probeBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME,
+ Types.LONG,
+ Types.LIST(rightType)));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ PROBE_BUFFER_STATE_NAME, Types.LONG, leftType));
+ probeBufferSeq =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(
+ PROBE_BUFFER_SEQ_STATE_NAME, Types.LONG));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice. Both timer kinds use VoidNamespace; they are told apart by timer type
+ // (event-time -> flip drain, processing-time -> TTL eviction), not by namespace.
+ timerService =
+ getInternalTimerService(TIMER_SERVICE_NAME, VoidNamespaceSerializer.INSTANCE, this);
+
+ // RocksDB/ForSt iterate map entries in serialized-key order, so the row-time-keyed build
+ // change buffer is scanned in event-time order; unordered backends (heap) need a sort.
+ final String backendId = getKeyedStateBackend().getBackendTypeIdentifier();
+ sortedStateBackend =
+ StateBackendLoader.ROCKSDB_STATE_BACKEND_NAME.equals(backendId)
+ || StateBackendLoader.FORST_STATE_BACKEND_NAME.equals(backendId);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(NUM_STATE_TTL_EVICTIONS_METRIC_NAME);
+ numUnmatchedProbeRecords = metricGroup.counter(NUM_UNMATCHED_PROBE_METRIC_NAME);
+ numUnmatchedBuildRetractions =
+ metricGroup.counter(NUM_UNMATCHED_BUILD_RETRACTIONS_METRIC_NAME);
+
+ // Best-effort in-memory tally; not restored, so it starts at 0 after a restore/rescale.
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(NUM_PROBE_BUFFERED_METRIC_NAME, probeBufferedGauge);
+ metricGroup.gauge(CURRENT_BUILD_WM_METRIC_NAME, buildWmGauge);
+ metricGroup.gauge(CURRENT_PROBE_WM_METRIC_NAME, probeWmGauge);
+ metricGroup.gauge(MAX_JOIN_FAN_OUT_METRIC_NAME, maxFanOutGauge);
+ metricGroup.gauge(AVG_JOIN_FAN_OUT_METRIC_NAME, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(CURRENT_PHASE_METRIC_NAME, phaseGauge);
+
+ // Pin the build-side input idle: the operator absorbs build-side watermarks and status.
+ combinedWatermark.updateStatus(1, true);
+
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ // Buffer during LOAD; in JOIN, buffer too if this key's flip drain has not finished yet
+ // (probeBufferSeq != null), so the new probe is joined in order after the buffered ones by
+ // its own timer instead of draining the whole buffer inline. Otherwise join immediately.
+ if (phase == Phase.LOAD || probeBufferSeq.value() != null) {
+ bufferProbe(probe);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ /**
+ * Buffers a probe row for the current key and registers a per-record event-time flip timer. The
+ * synthetic timestamp {@code (seq + 1) - Long.MAX_VALUE} is negative (so any non-negative
+ * watermark fires it) and increases with the sequence number (so timers drain in insertion
+ * order on every state backend).
+ */
+ private void bufferProbe(RowData probe) throws Exception {
+ long seq = probeBufferSeq.value() == null ? 0L : probeBufferSeq.value();
+ long ts = (seq + 1) - Long.MAX_VALUE;
+ probeBuffer.put(ts, probe);
+ probeBufferSeq.update(seq + 1);
+ timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, ts);
+ probeBufferedGauge.update(++probeBufferedCount);
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the change under its build row-time (grouping changes that share a row-time).
+ long rowtime = build.getLong(buildRowtimeIndex);
+ List<RowData> atRowtime = buildChangeBuffer.get(rowtime);
+ if (atRowtime == null) {
+ atRowtime = new ArrayList<>();
+ }
+ atRowtime.add(build);
+ buildChangeBuffer.put(rowtime, atRowtime);
+ // Tag the buffer with the current build watermark so it drains once the watermark passes.
+ // The tag may move backwards on restore (currentBuildSideWm resets to MIN_VALUE), which
+ // re-anchors a recovered buffer to this subtask's watermark.
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed; combined watermark is probe-driven.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // LOAD emits nothing downstream; track partial[0]'s idle bit for use after the flip.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only event-time (flip) timers are registered: each drains exactly one buffered probe for
+ // its key. All of a key's timers become due at the flip; interruptible timers fire them one
+ // at a time, yielding to checkpoints between firings.
+ long ts = timer.getTimestamp();
+ // Apply this key's due build-side changes before joining, so probes see the materialized
+ // build snapshot. Idempotent across the key's firings within one watermark sweep.
+ applyDueBufferedChanges();
+ RowData probe = probeBuffer.get(ts);
+ if (probe == null) {
+ // Already drained or evicted (e.g. by TTL) before this timer fired.
+ return;
+ }
+ joinProbeRow(probe);
+ probeBuffer.remove(ts);
+ probeBufferedCount = Math.max(0, probeBufferedCount - 1);
+ probeBufferedGauge.update(probeBufferedCount);
+ // This is the last buffered row iff it carries the highest sequence number; clearing the
+ // sequence here marks the buffer empty (cheap point lookup, no isEmpty() scan).
+ Long seq = probeBufferSeq.value();
+ if (seq != null && ts + Long.MAX_VALUE == seq) {
+ probeBufferSeq.clear();
+ }
+ }
+
+ /** Removes all probe-buffer state for the current key. */
+ private void clearProbeBuffer() {
+ probeBuffer.clear();
+ probeBufferSeq.clear();
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Only processing-time (TTL) timers are registered. Semantics match Flink's standard
+ // StateTtlConfig.
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // Reschedule if still in LOAD, or in JOIN but within stateTtlMs of the flip (grace window).
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(VoidNamespace.INSTANCE, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ clearAllPerKeyState();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN. Runs in a NON-KEYED context (callers {@link #processWatermark2}
+ * and {@link #idleFlipTimer}), so it must not touch keyed state directly; the per-key probe
+ * drain happens in the per-record event-time timers fired below.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Anchor the TTL grace window at the flip so keys loaded before it aren't evicted early.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ cancelIdleFlipTimer();
+
+ // If we have seen a (non-negative) probe-side WM, forward it so the per-record flip timers
+ // fire (their synthetic timestamps are negative) and drain the buffered probes. Firing is
+ // interruptible and the watermark is forwarded after the buffered probes are joined.
+ // If we haven't seen a probe-side WM yet, draining is triggered by the next probe-side WM.
+ if (currentProbeSideWm >= 0) {
+ super.processWatermark1(new Watermark(currentProbeSideWm));
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm >= 0 ? currentProbeSideWm : "none");
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // The build-side wm advanced: apply the changes that are now due (their row-time has
+ // been reached by the build-side watermark) and keep any not-yet-due changes buffered.
+ applyDueBufferedChanges();
+ return bufferedAtWmState.value();
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark seen by this subtask (recovery/rescale,
+ // or idle-timeout flip). Force-apply everything now, since there is no watermark to
+ // gate on and the next build watermark may not arrive for a long time.
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Applies the buffered build-side changes that are due — i.e. whose row-time has been reached
+ * by the build-side watermark — in event-time order, leaving any not-yet-due changes untouched
+ * in state.
+ */
+ private void applyDueBufferedChanges() throws Exception {
+ if (currentBuildSideWm == Long.MIN_VALUE) {
+ applyBufferedChanges();
+ return;
+ }
+ // Scan keys (row-times) only; the values (rows) of not-yet-due buckets stay serialized.
+ List<Long> due = new ArrayList<>();
+ boolean hasKept = false;
+ for (Long rowtime : buildChangeBuffer.keys()) {
Review Comment:
Here, we build an iterator and collect the keys;
Later, we iterate over these keys and access the entries again.
Can't we re-organize the code to avoid unnecessary state accesses?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3483328719
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
Review Comment:
> I'm not using the entry iterator to ensure that the buffered rows are read in the order in which they were inserted. RocksDBStateBackend would give us that order (due to the encoding of the long index values) but HeapStateBackend uses HashMap under the hood which doesn't preserve the order.
This can be detected via `KeyedStateStore.getBackendTypeIdentifier`.
But it would require two branches (for sorting and not-sorting backends)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on PR #28329:
URL: https://github.com/apache/flink/pull/28329#issuecomment-4633722953
⚠️ PR is not ready for review yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3437217193
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperatorTest.java:
##########
@@ -0,0 +1,1666 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.KeyedTwoInputStreamOperatorTestHarness;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.data.writer.BinaryRowWriter;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.Phase;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.runtime.util.RowDataHarnessAssertor;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.flink.types.RowKind;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_CHANGE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.BUILD_TABLE_STATE_NAME;
+import static org.apache.flink.table.runtime.operators.join.snapshot.LateralSnapshotJoinOperator.PROBE_BUFFER_STATE_NAME;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.deleteRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.insertRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateAfterRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.updateBeforeRecord;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.within;
+
+/** Harness tests for {@link LateralSnapshotJoinOperator}. */
+class LateralSnapshotJoinOperatorTest {
+
+ // ----------------------------------------------------------------- Schema
+
+ /** Probe row schema: (id BIGINT, key VARCHAR, val VARCHAR). */
+ private static final InternalTypeInfo<RowData> PROBE_TYPE =
+ InternalTypeInfo.ofFields(
+ new BigIntType(), VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+
+ /** Build row schema: (key VARCHAR, val VARCHAR, rt BIGINT). */
+ private static final InternalTypeInfo<RowData> BUILD_TYPE =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE, VarCharType.STRING_TYPE, new BigIntType());
+
+ /** Joined output schema: probe ++ build = (id, pKey, pVal, bKey, bVal, bRt). */
+ private static final LogicalType[] OUTPUT_TYPES = {
+ new BigIntType(),
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType()
+ };
+
+ /** Probe key column index (key VARCHAR is at field 1). */
+ private static final int PROBE_KEY_IDX = 1;
+
+ /** Build key column index (key VARCHAR is at field 0). */
+ private static final int BUILD_KEY_IDX = 0;
+
+ /** Build row-time column index (rt BIGINT is at field 2). */
+ private static final int BUILD_RT_IDX = 2;
+
+ private static final InternalTypeInfo<RowData> KEY_TYPE =
+ InternalTypeInfo.ofFields(VarCharType.STRING_TYPE);
+
+ private static final KeySelector<RowData, RowData> PROBE_KEY_SELECTOR =
+ nullSafeStringKeySelector(PROBE_KEY_IDX);
+ private static final KeySelector<RowData, RowData> BUILD_KEY_SELECTOR =
+ nullSafeStringKeySelector(BUILD_KEY_IDX);
+
+ private static final RowDataHarnessAssertor JOINED_ASSERTOR =
+ new RowDataHarnessAssertor(OUTPUT_TYPES);
+
+ // ----------------------------------------------------------------- Join conditions
+
+ /** Trivial join condition that always matches (equality is enforced by partitioning). */
+ private static final String ALWAYS_TRUE_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionStub extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionStub(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) { return true; }\n"
+ + "}\n";
+
+ /**
+ * Join condition that only matches when the probe value (field 2) equals {@code "match"}. Used
+ * to verify that the codegen'd condition is actually invoked at join time.
+ */
+ private static final String MATCH_VAL_JOIN_FUNC_CODE =
+ "public class LateralSnapshotJoinConditionMatchVal extends "
+ + "org.apache.flink.api.common.functions.AbstractRichFunction "
+ + "implements org.apache.flink.table.runtime.generated.JoinCondition {\n"
+ + " public LateralSnapshotJoinConditionMatchVal(Object[] reference) {}\n"
+ + " @Override public boolean apply("
+ + " org.apache.flink.table.data.RowData in1,"
+ + " org.apache.flink.table.data.RowData in2) {\n"
+ + " if (in1.isNullAt(2)) { return false; }\n"
+ + " return \"match\".equals(in1.getString(2).toString());\n"
+ + " }\n"
+ + "}\n";
+
+ private static GeneratedJoinCondition newTrueCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionStub", ALWAYS_TRUE_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ private static GeneratedJoinCondition newMatchValCondition() {
+ return new GeneratedJoinCondition(
+ "LateralSnapshotJoinConditionMatchVal", MATCH_VAL_JOIN_FUNC_CODE, new Object[0]);
+ }
+
+ // ----------------------------------------------------------------- Operator / harness
+ // factories
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ GeneratedJoinCondition joinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return new LateralSnapshotJoinOperator(
+ isLeftOuterJoin,
+ PROBE_TYPE,
+ BUILD_TYPE,
+ BUILD_RT_IDX,
+ joinCondition,
+ filterNullKeys,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static LateralSnapshotJoinOperator newOperator(
+ boolean isLeftOuterJoin,
+ Long loadCompletedTime,
+ Long loadCompletedIdleTimeoutMs,
+ Long stateTtlMs) {
+
+ return newOperator(
+ isLeftOuterJoin,
+ newTrueCondition(),
+ new boolean[] {true},
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ stateTtlMs);
+ }
+
+ private static KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData>
+ newHarness(LateralSnapshotJoinOperator op) throws Exception {
+ return new KeyedTwoInputStreamOperatorTestHarness<>(
+ op, PROBE_KEY_SELECTOR, BUILD_KEY_SELECTOR, KEY_TYPE);
+ }
+
+ private static KeySelector<RowData, RowData> nullSafeStringKeySelector(final int keyIdx) {
+ return value -> {
+ BinaryRowData ret = new BinaryRowData(1);
+ BinaryRowWriter writer = new BinaryRowWriter(ret);
+ if (value.isNullAt(keyIdx)) {
+ writer.setNullAt(0);
+ } else {
+ writer.writeString(0, value.getString(keyIdx));
+ }
+ writer.complete();
+ return ret;
+ };
+ }
+
+ // ----------------------------------------------------------------- LOAD phase
+
+ @Test
+ void loadPhaseBuildSideChangeProcessing() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // During LOAD, build-side changes are buffered and later applied in event-time order.
+ // -D for a never-inserted (key, value) pair is defensively ignored.
+ addBuildChange(h, deleteRecord("k1", "ghost", 5L));
+ // Two identical records (same key/val/row-time) → count(k1, v1, 20) = 2.
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ addBuildChange(h, insertRecord("k1", "v1", 20L));
+ // Earlier row-time than v1, but arrives later.
+ addBuildChange(h, insertRecord("k1", "v2", 10L));
+
+ // Still LOAD: changes are buffered, nothing applied or emitted yet.
+ assertPhase(op, Phase.LOAD);
+ assertThat(h.getOutput()).isEmpty();
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(buildTableKeys(h)).isEmpty();
+
+ // Advance the build watermark (still below the flip point) and access k1 again. The
+ // access drains the buffered batch in event-time order before buffering the new change.
+ addBuildWm(h, 50L);
+ addBuildChange(h, insertRecord("k1", "v3", 30L));
+ // TODO: also add updateBefore and updateAfter changes, to ensure that these are handled
+ // correctly
+
+ assertPhase(op, Phase.LOAD);
+ // Applied in row-time order: -D(ghost)@5 (ignored), +I(v2)@10, +I(v1)@20 ×2.
+ assertThat(buildTableKeys(h)).containsExactly("k1");
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L));
+ // The triggering v3 change is now buffered, awaiting the next drain.
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(1);
+ }
+ }
+
+ @Test
+ void loadPhaseProbeSideInputProcessing() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+
+ addBuildChange(h, insertRecord("k1", "build1", 1L));
+ addBuildWm(h, 20L);
+ addProbeRecord(h, 1L, "k1", "probe-load-1");
+ addProbeRecord(h, 2L, "k1", "probe-load-2");
+ addProbeWm(h, 50L);
+
+ assertPhase(op, Phase.LOAD);
+ // No output (records buffered, watermarks held back).
+ assertThat(h.getOutput()).isEmpty();
+
+ assertThat(op.getCurrentProbeSideWm()).isEqualTo(50L);
+ assertThat(probeBufferKeys(h)).containsExactly("k1");
+ assertThat(probeBufferForKey(h, op, "k1")).hasSize(2);
+ // Build-side change was applied to the build table.
+ assertThat(bufferedChangesForKey(h, op, "k1")).isEmpty();
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("build1", 1L));
+ }
+ }
+
+ // ----------------------------------------------------------------- Flip / transition
+
+ @ParameterizedTest(name = "leftOuter={0}, wmFlip={1}")
+ @CsvSource({"true, true", "true, false", "false, true", "false, false"})
+ void flipDrainsProbeBufferAndJoins(boolean leftOuter, boolean wmFlip) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(leftOuter, 100L, 200L, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ // LOAD: build state and buffered probes (one without a matching build row).
+ // k1 is a multi-set: count(v1)=2, count(v2)=1; k2 is a single row.
+ addBuildChange(h, insertRecord("k1", "build-k1-v1", 1L));
+ addBuildChange(h, insertRecord("k1", "build-k1-v1", 1L));
+ addBuildChange(h, insertRecord("k1", "build-k1-v2", 1L));
+ addBuildChange(h, insertRecord("k2", "build-k2", 1L));
+ // LOAD: add probe-side records
+ addProbeRecord(h, 1L, "k1", "probe-1");
+ addProbeRecord(h, 2L, "k2", "probe-2");
+ addProbeRecord(h, 3L, "k2", "probe-3");
+ addProbeRecord(h, 4L, "k3", "probe-no-match");
+ addProbeWm(h, 80L);
+ assertPhase(op, Phase.LOAD);
+
+ // assert that probe-side buffer is filled
+ assertThat(probeBufferForKey(h, op, "k1")).hasSize(1);
+ assertThat(probeBufferForKey(h, op, "k2")).hasSize(2);
+ assertThat(probeBufferForKey(h, op, "k3")).hasSize(1);
+ // idle-timeout timer is armed while in LOAD
+ assertThat(op.isIdleFlipTimerActive()).isTrue();
+
+ // trigger flip from LOAD to JOIN
+ if (wmFlip) {
+ // build WM crosses loadCompletedTime.
+ addBuildWm(h, 100L);
+ } else {
+ // proc-time exceeds idle timeout
+ h.setProcessingTime(200);
+ }
+ assertPhase(op, Phase.JOIN);
+ // idle-timeout timer is removed on flip (canceled by a WM flip, fired by an idle flip)
+ assertThat(op.isIdleFlipTimerActive()).isFalse();
+
+ // probe k1 joins k1's multi-set (count-respecting: 2x v1 + 1x v2 = three rows);
+ // probe k2 (2 rows) joins a single k2 build row;
+ // probe k3 doesn't have a matching build row. INNER: no output, LEFT OUTER: null-padded
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 80L);
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v2", 1L),
+ row(2L, "k2", "probe-2", "k2", "build-k2", 1L),
+ row(3L, "k2", "probe-3", "k2", "build-k2", 1L),
+ row(4L, "k3", "probe-no-match", null, null, null));
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v1", 1L),
+ row(1L, "k1", "probe-1", "k1", "build-k1-v2", 1L),
+ row(2L, "k2", "probe-2", "k2", "build-k2", 1L),
+ row(3L, "k2", "probe-3", "k2", "build-k2", 1L));
+ }
+ // Probe buffer drained on flip; build table preserved (k1 keeps multi-set counts).
+ assertThat(probeBufferKeys(h)).isEmpty();
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(
+ Map.of("build-k1-v1", 2L, "build-k1-v2", 1L));
+ assertThat(buildTableForKey(h, op, "k2"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("build-k2", 1L));
+ }
+ }
+
+ @Test
+ void idleTimerRearmsOnBuildWatermark() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 1000L, 100L, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(10);
+ h.open();
+ h.setProcessingTime(60);
+ // Build WM advances → re-arm.
+ addBuildWm(h, 10L);
+ // Original idle deadline was 10+100=110. Re-armed to 60+100=160.
+ h.setProcessingTime(159);
+ assertPhase(op, Phase.LOAD);
+ h.setProcessingTime(160);
+ assertPhase(op, Phase.JOIN);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void flipJoiningInvokesCodeGeneratedJoinCondition(boolean leftOuter) throws Exception {
+ LateralSnapshotJoinOperator op =
+ newOperator(
+ leftOuter, newMatchValCondition(), new boolean[] {true}, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addProbeRecord(h, 1L, "k1", "match");
+ addProbeRecord(h, 2L, "k1", "skip");
+ addProbeWm(h, 120L);
+ addBuildWm(h, 100L);
+
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 120L);
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "match", "k1", "v1", 1L),
+ row(2L, "k1", "skip", null, null, null));
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, "k1", "match", "k1", "v1", 1L));
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void flipJoiningCompositeEquiKeys(boolean leftOuter) throws Exception {
+ // Probe schema (kA VARCHAR, kB VARCHAR, val VARCHAR); build schema additionally carries a
+ // row-time attribute (kA VARCHAR, kB VARCHAR, val VARCHAR, rt BIGINT) at index 3.
+ InternalTypeInfo<RowData> probeType =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE, VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+ InternalTypeInfo<RowData> buildType =
+ InternalTypeInfo.ofFields(
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType());
+ final int buildRowtimeIndex = 3;
+ // Compose the composite key as a BinaryRowData with both key fields.
+ KeySelector<RowData, RowData> selector =
+ value -> {
+ BinaryRowData ret = new BinaryRowData(2);
+ BinaryRowWriter writer = new BinaryRowWriter(ret);
+ if (value.isNullAt(0)) {
+ writer.setNullAt(0);
+ } else {
+ writer.writeString(0, value.getString(0));
+ }
+ if (value.isNullAt(1)) {
+ writer.setNullAt(1);
+ } else {
+ writer.writeString(1, value.getString(1));
+ }
+ writer.complete();
+ return ret;
+ };
+ InternalTypeInfo<RowData> compositeKeyType =
+ InternalTypeInfo.ofFields(VarCharType.STRING_TYPE, VarCharType.STRING_TYPE);
+
+ LateralSnapshotJoinOperator op =
+ new LateralSnapshotJoinOperator(
+ leftOuter,
+ probeType,
+ buildType,
+ buildRowtimeIndex,
+ newTrueCondition(),
+ new boolean[] {true, true},
+ 100L,
+ null,
+ null);
+
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ new KeyedTwoInputStreamOperatorTestHarness<>(
+ op, selector, selector, compositeKeyType)) {
+ h.open();
+ // build rows: two identical rows for key a-1, a different row for key a-1, one row for
+ // key a-2.
+ addBuildChange(h, insertRecord("a", "1", "b-a-1-1", 1L));
+ addBuildChange(h, insertRecord("a", "1", "b-a-1-1", 1L));
+ addBuildChange(h, insertRecord("a", "1", "b-a-1-2", 1L));
+ addBuildChange(h, insertRecord("a", "2", "b-a-2-1", 1L));
+ // probes: matching composite, non-matching composite.
+ h.processElement1(insertRecord("a", "1", "p-a-1"));
+ h.processElement1(insertRecord("a", "9", "p-a-9"));
+ h.processElement1(insertRecord("b", "1", "p-b-1"));
+ h.processElement1(insertRecord("b", "9", "p-b-9"));
+ // flip to JOIN phase
+ addBuildWm(h, 100L);
+
+ LogicalType[] outTypes = {
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ VarCharType.STRING_TYPE,
+ new BigIntType()
+ };
+ RowDataHarnessAssertor compositeAssertor = new RowDataHarnessAssertor(outTypes);
+
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ compositeAssertor.shouldEmitAll(
+ h,
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-2", 1L),
+ compKeyRow("a", "9", "p-a-9", null, null, null, null),
+ compKeyRow("b", "1", "p-b-1", null, null, null, null),
+ compKeyRow("b", "9", "p-b-9", null, null, null, null));
+ } else {
+ compositeAssertor.shouldEmitAll(
+ h,
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-1", 1L),
+ compKeyRow("a", "1", "p-a-1", "a", "1", "b-a-1-2", 1L));
+ }
+ }
+ }
+
+ // ----------------------------------------------------------------- JOIN phase
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void joinPhaseImmediateInnerJoin(boolean leftOuter) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(leftOuter, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // First probe WM
+ addProbeWm(h, 10L);
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 10L);
+
+ // First probe — joined immediately.
+ addProbeRecord(h, 1L, "k1", "probe-immediate-1");
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, "k1", "probe-immediate-1", "k1", "v1", 1L));
+
+ // Another probe WM
+ addProbeWm(h, 20L);
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 20L);
+
+ // Second probe for same key — joined immediately.
+ addProbeRecord(h, 2L, "k1", "probe-immediate-2");
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(h, row(2L, "k1", "probe-immediate-2", "k1", "v1", 1L));
+
+ // Probe for non-existent key — no output (INNER).
+ addProbeRecord(h, 3L, "k2", "probe-no-match");
+ stripWatermarksAndStatusesFromOutput(h);
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(3L, "k2", "probe-no-match", null, null, null));
+ } else {
+ assertThat(h.extractOutputStreamRecords()).isEmpty();
+ }
+
+ // one more probe WM
+ addProbeWm(h, 30L);
+ assertWatermarkForwardedAfterRecords(h.getOutput(), 30L);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void joinPhaseBuildSideChangeApplication(boolean appliedByBuild) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // Two identical records for k1 (count 2) at row-time 1, buffered during LOAD.
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // Buffer four changes for k1
+ addBuildChange(h, deleteRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v2", 102L));
+ addBuildChange(h, updateBeforeRecord("k1", "v1", 1L));
+ addBuildChange(h, updateAfterRecord("k1", "v3", 103L));
+ // Buffer one change for k2.
+ addBuildChange(h, insertRecord("k2", "v1", 101L));
+
+ // assert number of buffered changes
+ assertThat(op.getCurrentBuildSideWm()).isEqualTo(100L);
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(bufferedAtWmFor(h, op, "k2")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k2")).hasSize(1);
+ assertThat(buildTableForKey(h, op, "k1")).isEqualTo(Map.of("v1", 2L));
+
+ // Probe record with no build WM advance - changes are not applied yet
+ addProbeRecord(h, 1L, "k1", "p-1");
+ JOINED_ASSERTOR.shouldEmitAll(
+ h, row(1L, "k1", "p-1", "k1", "v1", 1L), row(1L, "k1", "p-1", "k1", "v1", 1L));
+
+ // increment build-side WM
+ addBuildWm(h, 110L);
+
+ // assert that all changes are still buffered
+ assertThat(op.getCurrentBuildSideWm()).isEqualTo(110L);
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(4);
+ assertThat(bufferedAtWmFor(h, op, "k2")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k2")).hasSize(1);
+
+ // trigger application of k1 changes by build or probe-side input
+ if (!appliedByBuild) {
+ addBuildChange(h, insertRecord("k1", "v4", 111L));
+ // assert that changes have been applied and removed from buffer
+ // the triggering change is appended to the buffer
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(110L);
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(1);
+ assertThat(buildTableForKey(h, op, "k1")).isEqualTo(Map.of("v2", 1L, "v3", 1L));
+ // assert empty output
+ assertThat(h.getOutput()).isEmpty();
+ } else {
+ addProbeRecord(h, 2L, "k1", "p-2");
+ // assert that changes have been applied and removed from buffer
+ assertThat(bufferedAtWmFor(h, op, "k1")).isNull();
+ assertThat(bufferedChangesForKey(h, op, "k1")).isEmpty();
+ assertThat(buildTableForKey(h, op, "k1"))
+ .isEqualTo(Map.of("v2", 1L, "v3", 1L)); // /
+ // assert join results (v2 carries row-time 2, v3 carries row-time 1)
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(2L, "k1", "p-2", "k1", "v2", 102L),
+ row(2L, "k1", "p-2", "k1", "v3", 103L));
+ }
+
+ // assert that k2 change is still buffered
+ assertThat(bufferedAtWmFor(h, op, "k2")).isEqualTo(100L);
+ assertThat(bufferedChangesForKey(h, op, "k2")).hasSize(1);
+ // apply k2 change and join
+ addProbeRecord(h, 3L, "k2", "p-3");
+ assertThat(bufferedAtWmFor(h, op, "k2")).isNull();
+ assertThat(bufferedChangesForKey(h, op, "k2")).isEmpty();
+ assertThat(buildTableForKey(h, op, "k2")).isEqualTo(Map.of("v1", 1L));
+ // assert join results
+ JOINED_ASSERTOR.shouldEmitAll(h, row(3L, "k2", "p-3", "k2", "v1", 101L));
+ }
+ }
+
+ @Test
+ void joinPhaseWmForwardingLogic() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L); // flip
+ assertPhase(op, Phase.JOIN);
+
+ // Build-side WMs after flip are not forwarded.
+ addBuildWm(h, 200L);
+ assertThat(extractWatermarks(h.getOutput())).isEmpty();
+
+ // Probe-side WMs in JOIN are forwarded.
+ addProbeWm(h, 150L);
+ assertThat(extractWatermarks(h.getOutput())).containsExactly(new Watermark(150));
+ h.getOutput().clear();
+
+ // another build-side WM
+ addBuildWm(h, 300L);
+ assertThat(extractWatermarks(h.getOutput())).isEmpty();
+
+ addProbeWm(h, 250L);
+ assertThat(extractWatermarks(h.getOutput())).containsExactly(new Watermark(250));
+ }
+ }
+
+ /**
+ * For buffered build-side changes sharing the same row-time, retractions are applied before
+ * accumulations.
+ */
+ @Test
+ void joinPhaseAppliesRetractionsBeforeAccumulationsAtEqualRowTime() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // Drive into JOIN with empty build state for k1/k2.
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // Buffer accumulation-first at row-time 105 on absent rows:
+ // k1: +I then -D ; k2: +U then -U ; k3: +U then -U
+ addBuildChange(h, insertRecord("k1", "v1", 105L));
+ addBuildChange(h, deleteRecord("k1", "v1", 105L));
+ addBuildChange(h, updateAfterRecord("k2", "v1", 105L));
+ addBuildChange(h, updateBeforeRecord("k2", "v1", 105L));
+
+ // Advance build WM, then access each key to drain its buffer in event-time order.
+ addBuildWm(h, 200L);
+ addProbeRecord(h, 1L, "k1", "p1");
+ addProbeRecord(h, 2L, "k2", "p2");
+
+ assertThat(buildTableForKey(h, op, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+ assertThat(buildTableForKey(h, op, "k2"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 105L),
+ row(2L, "k2", "p2", "k2", "v1", 105L));
+ }
+ }
+
+ // ----------------------------------------------------------------- NULL keys
+
+ @ParameterizedTest
+ @CsvSource({"true, true", "true, false", "false, true", "false, false"})
+ void joinRespectsNullKeysFilter(boolean leftOuter, boolean filterNullKey) throws Exception {
+ LateralSnapshotJoinOperator op =
+ newOperator(
+ leftOuter,
+ newTrueCondition(),
+ new boolean[] {filterNullKey},
+ 100L,
+ null,
+ null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addBuildChange(h, insertRecord(null, "v_null", 1L));
+ addProbeRecord(h, 1L, null, "p_null");
+ addBuildWm(h, 100L);
+
+ // test joining during LOAD -> JOIN transition
+ if (filterNullKey) {
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, null, "p_null", null, null, null));
+ } else {
+ assertThat(h.getOutput()).isEmpty();
+ }
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(1L, null, "p_null", null, "v_null", 1L));
+ }
+
+ // test joining in JOIN phase
+ addProbeRecord(h, 2L, null, "p_null");
+ if (filterNullKey) {
+ if (leftOuter) {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(2L, null, "p_null", null, null, null));
+ } else {
+ assertThat(h.getOutput()).isEmpty();
+ }
+ } else {
+ JOINED_ASSERTOR.shouldEmitAll(h, row(2L, null, "p_null", null, "v_null", 1L));
+ }
+ }
+ }
+
+ // ----------------------------------------------------------------- Watermark status
+
+ @Test
+ void buildSideWmAndWmStatusForwarding() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // probe-side WM
+ addProbeWm(h, 70L);
+
+ // LOAD phase: build-side becomes idle then active again. WM is advanced.
+ h.processWatermarkStatus2(WatermarkStatus.IDLE);
+ h.processWatermarkStatus2(WatermarkStatus.ACTIVE);
+ addBuildWm(h, 50L);
+ // assert that no WMs or WM statuses are emitted in LOAD.
+ assertThat(h.getOutput()).isEmpty();
+
+ // Drive into JOIN
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // assert that only the probe-side WM was forwarded after the transition
+ assertThat(h.getOutput()).containsExactly(new Watermark(70L));
+ h.getOutput().clear();
+
+ // JOIN phase: build-side becomes idle then active again. WM is advanced
+ h.processWatermarkStatus2(WatermarkStatus.IDLE);
+ h.processWatermarkStatus2(WatermarkStatus.ACTIVE);
+ addBuildWm(h, 200L);
+
+ // No records, WMs, or WM statuses emitted in JOIN.
+ assertThat(h.getOutput()).isEmpty();
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void probeSideWmAndWmStatusForwarding(boolean probeIdleDuringLoad) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ addProbeWm(h, 25L);
+ addProbeWm(h, 50L);
+ // Probe-side WM statuses received during LOAD.
+ h.processWatermarkStatus1(WatermarkStatus.IDLE);
+ if (!probeIdleDuringLoad) {
+ h.processWatermarkStatus1(WatermarkStatus.ACTIVE);
+ }
+ // Absorbed during LOAD — nothing emitted.
+ assertThat(extractWatermarks(h.getOutput())).isEmpty();
+ assertThat(extractWatermarkStatuses(h.getOutput())).isEmpty();
+
+ // Flip to JOIN phase — last probe WM emitted.
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ if (probeIdleDuringLoad) {
+ assertThat(extractWatermarkStatuses(h.getOutput()))
+ .containsExactly(WatermarkStatus.IDLE);
+ } else {
+ assertThat(extractWatermarkStatuses(h.getOutput())).isEmpty();
+ }
+ assertThat(extractWatermarks(h.getOutput())).containsExactly(new Watermark(50));
+ h.getOutput().clear();
+
+ // WMs and WM status updates received during JOIN are forwarded.
+ addProbeWm(h, 150L);
+ assertThat(h.getOutput()).containsExactly(new Watermark(150));
+ h.getOutput().clear();
+ // set probe-side to idle
+ h.processWatermarkStatus1(WatermarkStatus.IDLE);
+ assertThat(h.getOutput()).containsExactly(WatermarkStatus.IDLE);
+ h.getOutput().clear();
+ // set probe-side to active
+ h.processWatermarkStatus1(WatermarkStatus.ACTIVE);
+ assertThat(h.getOutput()).containsExactly(WatermarkStatus.ACTIVE);
+ h.getOutput().clear();
+ // emit another watermark
+ addProbeWm(h, 200L);
+ assertThat(h.getOutput()).containsExactly(new Watermark(200));
+ h.getOutput().clear();
+ }
+ }
+
+ // ----------------------------------------------------------------- State TTL
+
+ @Test
+ void stateTtlRefreshesOnAccessAndEvictsInactiveKeys() throws Exception {
+ // stateTtlMs = 100. Timers are registered at 1.5 × stateTtlMs, so the deadline for access
+ // at t=0 is 150.
+ LateralSnapshotJoinOperator op = newOperator(false, 50L, null, 100L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ addBuildChange(h, insertRecord("k3", "v1", 1L));
+ addBuildChange(h, insertRecord("k4", "v1", 1L));
+
+ // State is NOT evicted during LOAD even after the deadline passes (TTL fires are
+ // rescheduled past the LOAD phase).
+ h.setProcessingTime(200);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3", "k4");
+
+ // flip to JOIN
+ addBuildWm(h, 50L);
+ assertPhase(op, Phase.JOIN);
+
+ // Touch k1, k2, k3 to reset their TTL in JOIN; leave k4 alone.
+ h.setProcessingTime(275);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3", "k4");
+ addBuildChange(h, insertRecord("k1", "v2", 2L));
+ addBuildChange(h, insertRecord("k2", "v2", 2L));
+ addBuildChange(h, insertRecord("k3", "v2", 2L));
+
+ // k4 evicted: it wasn't accessed since proc-time (0) and we flipped to JOIN at (200)
+ h.setProcessingTime(350);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3");
+
+ // Update build state for k1 and k2; leave k3 alone
+ addBuildWm(h, 60L);
+ h.setProcessingTime(400);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2", "k3");
+ addBuildChange(h, insertRecord("k1", "v3", 3L));
+ addBuildChange(h, insertRecord("k2", "v3", 3L));
+
+ // k3 evicted, not accessed since (275)
+ addBuildWm(h, 70L);
+ h.setProcessingTime(475);
+ assertThat(buildStateKeys(h)).containsExactly("k1", "k2");
+ // Access k1 from probe side to reset its TTL again; leave k2 alone
+ addProbeRecord(h, 1L, "k1", "p1");
+
+ // k2 evicted, not accessed since (400)
+ h.setProcessingTime(550);
+ assertThat(buildStateKeys(h)).containsExactly("k1");
+
+ // k1 finally evicted.
+ h.setProcessingTime(700);
+ assertThat(buildStateKeys(h)).isEmpty();
+
+ // The probe joined against the (k1) multi-set state at proc-time 475 — entries v1,
+ // v2, v3.
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v2", 2L),
+ row(1L, "k1", "p1", "k1", "v3", 3L));
+ }
+ }
+
+ @Test
+ void stateTtlClearsAllPerKeyState() throws Exception {
+ // stateTtlMs = 100. Timers are registered at 1.5 × stateTtlMs.
+ LateralSnapshotJoinOperator op = newOperator(false, 50L, null, 100L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.setProcessingTime(0);
+ h.open();
+ // Buffered during LOAD
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 50L); // flip to JOIN
+ assertPhase(op, Phase.JOIN);
+
+ // Buffer a change in JOIN that does NOT drain (no further WM advance / access). This
+ // populates the change buffer and the buffered-at tag while the build table keeps {v1}.
+ addBuildChange(h, insertRecord("k1", "v2"));
+ assertThat(buildTableForKey(h, op, "k1")).isEqualTo(Map.of("v1", 1L));
+ assertThat(bufferedChangesForKey(h, op, "k1")).hasSize(1);
+ assertThat(bufferedAtWmFor(h, op, "k1")).isEqualTo(50L);
+ assertThat(ttlExpiryFor(h, op, "k1")).isEqualTo(150L);
+
+ // Fire the TTL timer (well past the eviction deadline).
+ h.setProcessingTime(1000);
+
+ // Every per-key state object is cleared, not just the build table.
+ assertThat(buildTableForKey(h, op, "k1")).isEmpty();
+ assertThat(bufferedChangesForKey(h, op, "k1")).isEmpty();
+ assertThat(bufferedAtWmFor(h, op, "k1")).isNull();
+ assertThat(probeBufferForKey(h, op, "k1")).isEmpty();
+ assertThat(ttlExpiryFor(h, op, "k1")).isNull();
+ }
+ }
+
+ @Test
+ void stateTtlRestoreResetsFlipProcTime() throws Exception {
+ // stateTtlMs = 50. Build write at t=0 arms TTL at 75. Flip at t=0 → flipProcTime=0.
+ // Original grace ends at 0+50=50.
+ LateralSnapshotJoinOperator op1 = newOperator(false, 100L, null, 50L);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.setProcessingTime(0);
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertThat(op1.getFlipProcTime()).isEqualTo(0L);
+ state = h.snapshot(0L, 0L);
+ }
+
+ // Restart at t=30 — flipProcTime is re-anchored to 30; new grace ends at 30+50=80.
+ LateralSnapshotJoinOperator op2 = newOperator(false, 100L, null, 50L);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.setProcessingTime(30);
+ h.initializeState(state);
+ h.open();
+ assertThat(op2.getFlipProcTime()).isEqualTo(30L);
+
+ // At t=75 the recovered TTL timer fires. Grace check: now=75 < flipProcTime+stateTtlMs
+ // = 80 → reschedule rather than evict.
+ h.setProcessingTime(75);
+
+ // k1 still present because the grace window was re-anchored.
+ assertThat(buildStateKeys(h)).containsExactly("k1");
+
+ // advance proc-time to 105 to evict state
+ h.setProcessingTime(105);
+ assertThat(buildStateKeys(h)).isEmpty();
+ }
+ }
+
+ // ----------------------------------------------------------------- Snapshot / restore
+
+ @Test
+ void restoreFromLoadPhaseSnapshot() throws Exception {
+ LateralSnapshotJoinOperator op1 = newOperator(false, 100L, null, null);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.open();
+ // Build-side multi-set with a duplicate count, plus a buffered
+ // probe.
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v2", 2L));
+ addBuildWm(h, 10L);
+ addBuildChange(h, insertRecord("k1", "v3", 3L));
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertPhase(op1, Phase.LOAD);
+ assertThat(probeBufferKeys(h)).containsExactly("k1");
+ state = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator op2 = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.initializeState(state);
+ h.open();
+ assertPhase(op2, Phase.LOAD);
+ // Buffered probe and build-table multi-set (with counts) preserved across restore.
+ assertThat(probeBufferKeys(h)).containsExactly("k1");
+ assertThat(buildTableForKey(h, op2, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L));
+ assertThat(bufferedChangesForKey(h, op2, "k1")).hasSize(1);
+ // Gauge is rebuilt from restored keyed state: one probe buffered for k1.
+ assertThat(op2.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(1L);
+
+ // Trigger flip; the buffered probe is joined post-restore, count-respecting against the
+ // restored multi-set (2× v1 + 1× v2 = three rows).
+ addProbeWm(h, 50L);
+ addBuildWm(h, 100L);
+ assertPhase(op2, Phase.JOIN);
+ assertThat(probeBufferKeys(h)).isEmpty();
+ // Draining the restored probe and build buffers brings both gauges back to 0.
+ assertThat(op2.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ assertThat(buildTableForKey(h, op2, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L, "v3", 1L));
+
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v2", 2L),
+ row(1L, "k1", "p1", "k1", "v3", 3L));
+ }
+ }
+
+ @Test
+ void restoreFromMixedPhaseSnapshot() throws Exception {
+ // Subtask A: drive into JOIN with a buffered change for k1.
+ LateralSnapshotJoinOperator opA = newOperator(false, 100L, null, null);
+ OperatorSubtaskState stateA;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(opA)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(opA, Phase.JOIN);
+ // Buffer a build-side change
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ stateA = h.snapshot(0L, 0L);
+ }
+
+ // Subtask B: stay in LOAD (no flip-triggering build WM).
+ LateralSnapshotJoinOperator opB = newOperator(false, 100L, null, null);
+ OperatorSubtaskState stateB;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(opB)) {
+ h.open();
+ addProbeRecord(h, 1L, "k2", "p1");
+ assertPhase(opB, Phase.LOAD);
+ stateB = h.snapshot(0L, 0L);
+ }
+
+ OperatorSubtaskState combined =
+ AbstractStreamOperatorTestHarness.repackageState(stateA, stateB);
+
+ // Restore the combined state — phase must be LOAD because some subtask was LOAD.
+ LateralSnapshotJoinOperator opC = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(opC)) {
+ h.initializeState(combined);
+ h.open();
+ assertPhase(opC, Phase.LOAD);
+
+ // assert state: A's build table ({v1:1}) plus A's buffered change and B's probe.
+ assertThat(buildTableKeys(h)).containsExactly("k1");
+ assertThat(buildTableForKey(h, opC, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+ assertThat(bufferedChangesForKey(h, opC, "k1")).hasSize(1);
+ assertThat(probeBufferKeys(h)).containsExactly("k2");
+
+ // During LOAD the current build WM is still MIN_VALUE, so the recovered buffer (tagged
+ // at the pre-restore WM) is not drained; the new change is appended to it.
+ addBuildChange(h, insertRecord("k1", "v2", 2L));
+ assertThat(bufferedChangesForKey(h, opC, "k1")).hasSize(2);
+ assertThat(buildTableForKey(h, opC, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 1L));
+
+ // Trigger flip
+ addBuildWm(h, 100L);
+ assertPhase(opC, Phase.JOIN);
+
+ // Access k1: the buffered changes drain in event-time order, then the probe joins.
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(bufferedChangesForKey(h, opC, "k1")).isEmpty();
+ assertThat(buildTableForKey(h, opC, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v1", 2L, "v2", 1L));
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v1", 1L),
+ row(1L, "k1", "p1", "k1", "v2", 2L));
+ }
+ }
+
+ @Test
+ void restoreFromJoinPhaseSnapshot() throws Exception {
+ LateralSnapshotJoinOperator op1 = newOperator(false, 100L, null, null);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.open();
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op1, Phase.JOIN);
+ // Buffer a -U/+U pair for each key (tagged at bufferedAt = 100).
+ addBuildChange(h, updateBeforeRecord("k1", "v1", 1L));
+ addBuildChange(h, updateAfterRecord("k1", "v2", 101L));
+ addBuildChange(h, updateBeforeRecord("k2", "v1", 1L));
+ addBuildChange(h, updateAfterRecord("k2", "v2", 101L));
+ state = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator op2 = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.initializeState(state);
+ h.open();
+ assertPhase(op2, Phase.JOIN);
+ assertThat(op2.getCurrentBuildSideWm()).isEqualTo(Long.MIN_VALUE);
+ assertThat(bufferedChangesForKey(h, op2, "k1")).hasSize(2);
+ assertThat(bufferedChangesForKey(h, op2, "k2")).hasSize(2);
+ // Gauge is rebuilt from restored keyed state: four build-side changes buffered
+ // (two for k1, two for k2).
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(4L);
+
+ // k2: accessed while no build WM has arrived since restore → eager drain.
+ addProbeRecord(h, 1L, "k2", "p-k2");
+ assertThat(bufferedChangesForKey(h, op2, "k2")).isEmpty();
+ // Draining k2's two restored changes leaves k1's two still buffered.
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(2L);
+ assertThat(buildTableForKey(h, op2, "k2"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v2", 1L));
+
+ // k1: still buffered (eager drain of k2 left latestBuildSideWm at MIN_VALUE); advance
+ // the build WM, then access → normal WM-gated drain.
+ assertThat(bufferedChangesForKey(h, op2, "k1")).hasSize(2);
+ addBuildWm(h, 200L);
+ addProbeRecord(h, 2L, "k1", "p-k1");
+ assertThat(bufferedChangesForKey(h, op2, "k1")).isEmpty();
+ // All restored changes drained; gauge back to 0.
+ assertThat(op2.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ assertThat(buildTableForKey(h, op2, "k1"))
+ .containsExactlyInAnyOrderEntriesOf(Map.of("v2", 1L));
+
+ stripWatermarksAndStatusesFromOutput(h);
+ JOINED_ASSERTOR.shouldEmitAll(
+ h,
+ row(1L, "k2", "p-k2", "k2", "v2", 101L),
+ row(2L, "k1", "p-k1", "k1", "v2", 101L));
+ }
+ }
+
+ @Test
+ void restoreRearmsIdleFlipTimer() throws Exception {
+ LateralSnapshotJoinOperator op1 = newOperator(false, 1000L, 100L, null);
+ OperatorSubtaskState state;
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op1)) {
+ h.setProcessingTime(50);
+ h.open();
+ assertPhase(op1, Phase.LOAD);
+ state = h.snapshot(0L, 0L);
+ }
+
+ LateralSnapshotJoinOperator op2 = newOperator(false, 1000L, 100L, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op2)) {
+ h.setProcessingTime(150);
+ h.initializeState(state);
+ h.open();
+ assertPhase(op2, Phase.LOAD);
+ // Re-armed at open()'s proc-time + idleTimeout = 150 + 100 = 250.
+ h.setProcessingTime(249);
+ assertPhase(op2, Phase.LOAD);
+ h.setProcessingTime(250);
+ assertPhase(op2, Phase.JOIN);
+ }
+ }
+
+ // ----------------------------------------------------------------- Metrics
+
+ @Test
+ void currentPhaseGaugeReflectsPhase() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // LOAD = ordinal 0.
+ assertThat(op.getCurrentPhaseGauge().getValue()).isEqualTo(0);
+
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+ // JOIN = ordinal 1.
+ assertThat(op.getCurrentPhaseGauge().getValue()).isEqualTo(1);
+ }
+ }
+
+ @Test
+ void probeBufferedGaugeTracksLoadBufferingAndDrainsOnFlip() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(1L);
+ addProbeRecord(h, 2L, "k2", "p2");
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(2L);
+ addProbeRecord(h, 3L, "k1", "p3");
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+
+ // Flip drains all buffered probes across keys.
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getNumProbeSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ }
+ }
+
+ @Test
+ void buildBufferedGaugeTracksJoinBufferingAndDrains() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // Flip to JOIN with an empty build state (no build records during LOAD).
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+
+ // JOIN-phase build changes are buffered (their tag equals the current build WM).
+ addBuildChange(h, insertRecord("k1", "v2", 101L));
+ addBuildChange(h, insertRecord("k1", "v3", 101L));
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(2L);
+ addBuildChange(h, insertRecord("k2", "v1", 102L));
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(3L);
+
+ // Advance build WM, then drain each key by accessing it from the probe side.
+ addBuildWm(h, 110L);
+ addProbeRecord(h, 1L, "k1", "p1");
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(1L);
+ addProbeRecord(h, 2L, "k2", "p2");
+ assertThat(op.getNumBuildSideRecordsBufferedGauge().getValue()).isEqualTo(0L);
+ }
+ }
+
+ @Test
+ void watermarkGaugesTrackBuildAndProbeWatermarks() throws Exception {
+ // High loadCompletedTime so the operator stays in LOAD for the first build WMs.
+ LateralSnapshotJoinOperator op = newOperator(false, 1000L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(Long.MIN_VALUE);
+ assertThat(op.getCurrentProbeSideWatermarkGauge().getValue()).isEqualTo(Long.MIN_VALUE);
+
+ // LOAD phase: both watermarks are tracked even though nothing is forwarded.
+ addBuildWm(h, 50L);
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(50L);
+ addProbeWm(h, 70L);
+ assertThat(op.getCurrentProbeSideWatermarkGauge().getValue()).isEqualTo(70L);
+ assertPhase(op, Phase.LOAD);
+
+ // Flip to JOIN.
+ addBuildWm(h, 1000L);
+ assertPhase(op, Phase.JOIN);
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(1000L);
+
+ // JOIN phase: probe WM gauge keeps tracking (guards the dedicated currentProbeWm
+ // field).
+ addProbeWm(h, 1500L);
+ assertThat(op.getCurrentProbeSideWatermarkGauge().getValue()).isEqualTo(1500L);
+ addBuildWm(h, 1100L);
+ assertThat(op.getCurrentBuildSideWatermarkGauge().getValue()).isEqualTo(1100L);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void fanOutGaugesTrackMaxAndAverage(boolean leftOuter) throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(leftOuter, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // k1 multi-set: v1 x2, v2 x1 → fan-out 3; k2 single row → fan-out 1.
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 1L));
+ addBuildChange(h, insertRecord("k1", "v2", 1L));
+ addBuildChange(h, insertRecord("k2", "v1", 1L));
+ // a probe record during load
+ addProbeRecord(h, 1L, "k2", "p1"); // fan-out 1
+ // a probe record that does not match
+ addProbeRecord(h, 2L, "k3", "p2"); // fan-out INNER: 0, LEFT OUTER: 1
+ addBuildWm(h, 100L);
+ assertPhase(op, Phase.JOIN);
+
+ // check merge stats after transition
+ assertThat(op.getMaxJoinFanOutGauge().getValue()).isEqualTo(1L);
+ if (leftOuter) {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isEqualTo(2d / 2);
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(0L);
+ } else {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(1d / 2, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(1L);
+ }
+
+ // join probe in JOIN phase
+ addProbeRecord(h, 3L, "k1", "p3"); // fan-out 3
+ assertThat(op.getMaxJoinFanOutGauge().getValue()).isEqualTo(3L);
+ if (leftOuter) {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isEqualTo(5d / 3, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(0L);
+ } else {
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(4d / 3, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(1L);
+ }
+
+ addProbeRecord(h, 3L, "k3", "no-match");
+ assertThat(op.getMaxJoinFanOutGauge().getValue()).isEqualTo(3L);
+ if (leftOuter) {
+ // fan-out 1 (LEFT OUTER, null-padded)
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(6.0d / 4, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(0L);
+ } else {
+ // fan-out 0 (INNER, unmatched)
+ assertThat(op.getAvgJoinFanOutGauge().getValue()).isCloseTo(4.0d / 4, within(1e-9));
+ assertThat(op.getNumUnmatchedProbeRecords().getCount()).isEqualTo(2L);
+ }
+ }
+ }
+
+ @Test
+ void numUnmatchedBuildRetractionsCountsAbsentRowRetractions() throws Exception {
+ LateralSnapshotJoinOperator op = newOperator(false, 100L, null, null);
+ try (KeyedTwoInputStreamOperatorTestHarness<RowData, RowData, RowData, RowData> h =
+ newHarness(op)) {
+ h.open();
+ // LOAD: a -D for a never-inserted row is counted.
+ addBuildChange(h, deleteRecord("k1", "ghost", 1L));
+ addBuildChange(h, insertRecord("k1", "v1", 2L));
+ assertThat(op.getNumUnmatchedBuildRetractions().getCount()).isEqualTo(0L);
Review Comment:
Two tests suggestions:
- **ref-count underflow**: right now only the metric is asserted - could we also check the multiset map itself, e.g. that a `-D` on a count-1 row removes the key rather than going negative?
- **atomic `-U`/`+U` visibility mid-buffer**: it'd be great to have a case where a probe joins *while* a `-U`/`+U` pair is still buffered (should see the old value), then re-probes after the watermark advances (sees the new value) - that's really the whole point of the buffer, so worth pinning down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3453604766
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
Review Comment:
The join only accepts append-only probe-side record.
So we don't have to deal with lateness due to retractions. If downstream operators aren't able to handle late records, they can drop them themselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3453697545
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
Review Comment:
The later state clean up doesn't touch all keyed states.
We can add a helper method, but IMO it's not going to make the code easier to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3453791511
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
Review Comment:
The operator does not join records on event-time but rather joins any probe-side record against the current (as of processing-time) build-side table.
We use event-time only to flip from LOAD to JOIN phase.
However, implicitly watermark source-alignment should help to keep build and probe-sides "roughly aligned" without paying the latency price of exact event-time processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3453741690
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
Review Comment:
Yes, that's OK. This join operates with processing-time semantics, so there is no single correct result anyway because the results depend on the order in which records arrive at the operator.
Buffering build-side changes prevents that we join against incomplete update changes (only -U/-D have been processed, and +U/+I haven't arrived yet).
This allows the join operator to join with much lower latency and without stalling because an idle build-side doesn't advance the WM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3453764499
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
Review Comment:
So far, I've kept the operator generic to not blowup the complexity even more.
In case of a PK-FK join, the key columns should be part of the key, so that should mean that `buildTableState` should just have a single row.
Or am I misunderstanding, what you are proposing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3476086529
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
Review Comment:
I've added a short paragraph to the class JavaDocs:
> Streaming only: The operator joins data with processing-time semantics which can't be (easily) done in batch. Also, the operator's phase is held in union operator state, which is incompatible with finished operators. Joins against LATERAL SNAPSHOT functions in batch queries should be translated to regular joins. The changes on the build-side would be consolidated into a final table and then be joined with the probe-side. All probe-side records are then joined against the same (and final) version of the build-side input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on PR #28329:
URL: https://github.com/apache/flink/pull/28329#issuecomment-4968940072
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3507368063
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
Review Comment:
I've adapted the implementation to also use MapState for the buildChangeBuffer`.
Please take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3554790359
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
+ buildTableState.put(change, currentCnt == null ? 1L : currentCnt + 1L);
+ } else {
+ // -D / -U
+ if (currentCnt == null || currentCnt <= 0L) {
+ numUnmatchedBuildRetractions.inc();
+ return;
+ }
Review Comment:
I think now that we're gating the changes-to-apply by watermark; it's a more explicit contract with the user/source that all the related changes show up until that WM.
One discrepancy though is that we keep late INSERTs but drop late DELETEs (I'm referring to [this](https://github.com/apache/flink/pull/28329/changes/BASE..a0f4723108667a4b19ea5f37fc0a7046e87201b3#r3547523930) comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3554790359
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
+ }
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ buildBufferedCount = Math.max(0, buildBufferedCount - changes.size());
+ buildBufferedGauge.update(buildBufferedCount);
+ }
+
+ /**
+ * Orders build-side changelog entries for deterministic, event-time-ordered application:
+ * ascending by the build-side row-time attribute, then retractions ({@code -U}/{@code -D})
+ * before accumulations ({@code +U}/{@code +I}) for entries sharing a row-time.
+ */
+ private int compareBuildChanges(RowData a, RowData b) {
+ int byTime = Long.compare(a.getLong(buildRowtimeIndex), b.getLong(buildRowtimeIndex));
+ if (byTime != 0) {
+ return byTime;
+ }
+ return Integer.compare(retractionRank(a), retractionRank(b));
+ }
+
+ /** Returns {@code 0} for retracting changes ({@code -U}/{@code -D}), {@code 1} otherwise. */
+ private static int retractionRank(RowData change) {
+ RowKind kind = change.getRowKind();
+ return (kind == RowKind.UPDATE_BEFORE || kind == RowKind.DELETE) ? 0 : 1;
+ }
+
+ /**
+ * Apply a build-side change record directly to the build-table multi-set.
+ *
+ * <p>MUTATES the input row's {@link RowKind} to {@link RowKind#INSERT} to normalize the key
+ * used for {@code buildTableState} lookups. The caller must not rely on the original kind after
+ * this call returns. The mutation avoids a per-record copy on a hot path.
+ */
+ private void applyBuildChange(RowData change) throws Exception {
+ RowKind changeType = change.getRowKind();
+ change.setRowKind(RowKind.INSERT);
+ Long currentCnt = buildTableState.get(change);
+ if (changeType == RowKind.INSERT || changeType == RowKind.UPDATE_AFTER) {
+ // +I / +U
+ buildTableState.put(change, currentCnt == null ? 1L : currentCnt + 1L);
+ } else {
+ // -D / -U
+ if (currentCnt == null || currentCnt <= 0L) {
+ numUnmatchedBuildRetractions.inc();
+ return;
+ }
Review Comment:
I think now that we're gating the changes-to-apply by watermark; it's a more explicit contract with the user/source that all the related changes show up until that WM.
One discrepancy though is that we keep late INSERTs but drop late DELETEs (related to [this](https://github.com/apache/flink/pull/28329#discussion_r3547523930) comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3500429392
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
Review Comment:
The chunk-based processing wouldn't really benefit from an entry iterator because we would still need to `remove()` individual items and the iterator would need to skip removed items that weren't GC'd yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39781][table] Add LateralSnapshotJoinOperator with two-phase LOAD/JOIN execution [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28329:
URL: https://github.com/apache/flink/pull/28329#discussion_r3500577424
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/snapshot/LateralSnapshotJoinOperator.java:
##########
@@ -0,0 +1,1038 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.snapshot;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.DefaultOpenContext;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.utils.JoinedRowData;
+import org.apache.flink.table.runtime.generated.GeneratedJoinCondition;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.operators.join.JoinConditionWithNullFilters;
+import org.apache.flink.table.runtime.operators.metrics.SimpleGauge;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ScheduledFuture;
+
+/**
+ * Stream operator implementing the {@code LATERAL SNAPSHOT} processing-time temporal table join.
+ *
+ * <p>The operator runs in two operator-wide phases that progress only forward, LOAD then JOIN: LOAD
+ * bootstraps the build-side table from its changelog up to {@code loadCompletedTime}, buffering
+ * probe rows meanwhile; JOIN then continuously joins probe rows against the materialized build-side
+ * state. How each input is handled depends on the current phase:
+ *
+ * <ul>
+ * <li>Build-side (input2 / right) changes are handled the same way in both phases: they are
+ * buffered in {@code buildChangeBuffer} and applied lazily to a per-key multi-set in {@code
+ * buildTableState} on the next per-key access (build- or probe-side) once the build-side
+ * watermark has advanced past the buffer's tag, or at the flip. The buffered changelog is
+ * applied in event-time order: changes are sorted by the build-side row-time attribute
+ * ({@code buildRowtimeIndex}), and for equal row-times retractions ({@code -U}/{@code -D})
+ * are applied before accumulations ({@code +U}/{@code +I}). Buffering until the watermark
+ * passes preserves atomic update visibility across {@code -U}/{@code +U} pairs in JOIN phase.
+ * <li>Probe-side (input1 / left) records are handled differently per phase. During LOAD they are
+ * buffered in {@code probeBuffer} until the configured flip point is reached on the
+ * build-side watermark, at which point a per-key event-time timer drains the buffered probes
+ * and joins them with the materialized build-side state. During JOIN they are joined
+ * immediately with the current build-side state.
+ * </ul>
+ *
+ * <p>Watermark forwarding rules:
+ *
+ * <ul>
+ * <li>Build-side watermarks are never forwarded downstream.
+ * <li>Probe-side watermarks are held back during LOAD and forwarded during JOIN phase.
+ * </ul>
+ *
+ * <p>The flip from LOAD to JOIN phase is triggered by either:
+ *
+ * <ul>
+ * <li>the build-side watermark reaching {@code loadCompletedTime} (event-time gate), or
+ * <li>the {@code loadCompletedIdleTimeoutMs} processing-time timer firing without any build-side
+ * watermark advance.
+ * </ul>
+ *
+ * <p>State TTL eviction happens during JOIN phase and is implemented with keyed processing-time
+ * timers (matching the semantics of Flink's standard {@code StateTtlConfig}).
+ */
+@Internal
+public class LateralSnapshotJoinOperator extends AbstractStreamOperator<RowData>
+ implements TwoInputStreamOperator<RowData, RowData, RowData>, Triggerable<RowData, String> {
+
+ // -------------------------- static final definitions --------------------------
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(LateralSnapshotJoinOperator.class);
+
+ /** operator state names */
+ private static final String OPERATOR_PHASE_STATE_NAME = "lateral-snapshot-phase";
+
+ /** keyed state names */
+ @VisibleForTesting static final String BUILD_TABLE_STATE_NAME = "build-table";
+
+ @VisibleForTesting static final String BUILD_CHANGE_BUFFER_STATE_NAME = "build-change-buffer";
+ @VisibleForTesting static final String BUFFERED_AT_WM_STATE_NAME = "buffered-at-wm";
+ @VisibleForTesting static final String PROBE_BUFFER_STATE_NAME = "probe-buffer";
+ private static final String TTL_EXPIRY_STATE_NAME = "ttl-expiry";
+
+ /** timer service and namespace names */
+ private static final String TIMER_SERVICE_NAME = "lateral-snapshot-timers";
+
+ @VisibleForTesting static final String NS_FLIP = "flip";
+ @VisibleForTesting static final String NS_TTL = "ttl";
+
+ /**
+ * Event-time timestamp at which the per-key {@code probeBuffer} flip join timer is registered.
+ * Any non-{@code MIN_VALUE} watermark advance fires it.
+ */
+ @VisibleForTesting static final long FLIP_JOIN_TIMER_TS = 1L;
+
+ /** Gauge: probe-side records currently buffered during LOAD. */
+ @VisibleForTesting static final String M_NUM_PROBE_BUFFERED = "numProbeSideRecordsBuffered";
+
+ /** Gauge: build-side changes currently buffered during JOIN. */
+ @VisibleForTesting static final String M_NUM_BUILD_BUFFERED = "numBuildSideRecordsBuffered";
+
+ /** Counter: keys evicted from state by TTL. */
+ @VisibleForTesting static final String M_NUM_STATE_TTL_EVICTIONS = "numStateTtlEvictions";
+
+ /** Gauge: latest build-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_BUILD_WM = "currentBuildSideWatermark";
+
+ /** Gauge: latest probe-side watermark observed. */
+ @VisibleForTesting static final String M_CURRENT_PROBE_WM = "currentProbeSideWatermark";
+
+ /** Gauge: current phase ordinal, 0 = LOAD, 1 = JOIN. */
+ @VisibleForTesting static final String M_CURRENT_PHASE = "currentPhase";
+
+ /** Gauge: max joined records emitted for a probe-side record. */
+ @VisibleForTesting static final String M_MAX_JOIN_FAN_OUT = "maxJoinFanOut";
+
+ /** Gauge: average joined records emitted per probe-side record. */
+ @VisibleForTesting static final String M_AVG_JOIN_FAN_OUT = "avgJoinFanOut";
+
+ /** Counter: probe-side records without a match. */
+ @VisibleForTesting static final String M_NUM_UNMATCHED_PROBE = "numUnmatchedProbeRecords";
+
+ /** Counter: build-side retractions for a row not present in state. */
+ @VisibleForTesting
+ static final String M_NUM_UNMATCHED_BUILD_RETRACTIONS = "numUnmatchedBuildRetractions";
+
+ /** Two-phase state machine. */
+ public enum Phase {
+ LOAD,
+ JOIN
+ }
+
+ /** What triggered the {@link Phase#LOAD} to {@link Phase#JOIN} flip (for logging). */
+ private enum FlipTrigger {
+ BUILD_WATERMARK,
+ IDLE_TIMEOUT
+ }
+
+ // -------------------------- constructor args --------------------------
+
+ private final boolean isLeftOuterJoin;
+ private final InternalTypeInfo<RowData> leftType;
+ private final InternalTypeInfo<RowData> rightType;
+
+ /** Field index of the build-side (right) row-time attribute. */
+ private final int buildRowtimeIndex;
+
+ private final GeneratedJoinCondition generatedJoinCondition;
+
+ /**
+ * Per-equi-key flag indicating whether rows with a NULL in that key position must be filtered
+ * before the join condition runs (SQL semantics: {@code NULL = NULL} is not true).
+ */
+ private final boolean[] filterNullKeys;
+
+ /**
+ * Timestamp at which the build-side watermark must arrive for the operator to flip from {@code
+ * LOAD} to JOIN.
+ */
+ private final Long loadCompletedTime;
+
+ /**
+ * Processing-time idle timeout duration (millis) on build-side watermarks. When configured, the
+ * operator flips to JOIN if no build-side watermark advance is seen for this duration.
+ */
+ @Nullable private final Long loadCompletedIdleTimeoutMs;
+
+ /**
+ * State TTL (millis) to clean up any keyed state during JOIN phase. We schedule TTL timers
+ * maxStateTtlMs ahead and check on minStateTtlMs before scheduling a new timer. This avoids
+ * rescheduling timers on every state access while still ensuring that keyed state is evicted
+ * after at most maxStateTtlMs of key inactivity during JOIN phase. If minStateTtlMs is set to
+ * 0, state TTL is disabled.
+ */
+ private final long minStateTtlMs;
+
+ private final long maxStateTtlMs;
+
+ // -------------------------- transient runtime fields --------------------------
+
+ private transient JoinConditionWithNullFilters joinCondition;
+ private transient GenericRowData nullPaddedBuild;
+ private transient TimestampedCollector<RowData> collector;
+
+ private transient InternalTimerService<String> timerService;
+
+ private transient Phase phase;
+
+ /**
+ * Processing-time wall clock at which the operator transitioned from {@link Phase#LOAD} to
+ * {@link Phase#JOIN}. {@code null} while still in LOAD. Used by the TTL handler to reschedule
+ * state TTL timers that fire too early.
+ */
+ @Nullable private transient Long flipProcTime;
+
+ /** Highest build-side watermark observed. */
+ private transient long currentBuildSideWm;
+
+ /** Latest probe-side watermark observed. */
+ private transient long currentProbeSideWm;
+
+ /** Non-keyed processing-time idle-flip timer. */
+ @Nullable private transient ScheduledFuture<?> idleFlipTimer;
+
+ // -------------------------- keyed state --------------------------
+
+ /** Build-side table as multi-set: row → reference count. */
+ private transient MapState<RowData, Long> buildTableState;
+
+ /** Buffer for build-side changes during JOIN to ensure atomic updates. */
+ private transient ListState<RowData> buildChangeBuffer;
+
+ /** Build-side watermark to ensure atomic application of build changes during JOIN. */
+ private transient ValueState<Long> bufferedAtWmState;
+
+ /** Buffer for probe-side records during LOAD. */
+ private transient ListState<RowData> probeBuffer;
+
+ /** Most recently registered TTL timer deadline; used to advance TTL timer. */
+ private transient ValueState<Long> ttlExpiryState;
+
+ // -------------------------- operator state --------------------------
+
+ private transient ListState<String> operatorPhaseState;
+
+ // -------------------------- metrics --------------------------
+
+ private transient Counter numStateTtlEvictions;
+ private transient Counter numUnmatchedProbeRecords;
+ private transient Counter numUnmatchedBuildRetractions;
+
+ private transient SimpleGauge<Long> probeBufferedGauge;
+ private transient SimpleGauge<Long> buildBufferedGauge;
+ private transient SimpleGauge<Long> buildWmGauge;
+ private transient SimpleGauge<Long> probeWmGauge;
+ private transient SimpleGauge<Long> maxFanOutGauge;
+ private transient SimpleGauge<Double> avgFanOutGauge;
+ private transient Gauge<Integer> phaseGauge;
+
+ /** Backing accumulators for the push-model gauges (in-memory, not persisted, best-effort). */
+ private transient long probeBufferedCount;
+
+ private transient long buildBufferedCount;
+ private transient long maxJoinFanOut;
+ private transient long totalJoinFanOut;
+ private transient long totalProbeJoins;
+
+ public LateralSnapshotJoinOperator(
+ boolean isLeftOuterJoin,
+ InternalTypeInfo<RowData> leftType,
+ InternalTypeInfo<RowData> rightType,
+ int buildRowtimeIndex,
+ GeneratedJoinCondition generatedJoinCondition,
+ boolean[] filterNullKeys,
+ Long loadCompletedTime,
+ @Nullable Long loadCompletedIdleTimeoutMs,
+ @Nullable Long stateTtlMs) {
+ this.isLeftOuterJoin = isLeftOuterJoin;
+ this.leftType = Preconditions.checkNotNull(leftType);
+ this.rightType = Preconditions.checkNotNull(rightType);
+ this.buildRowtimeIndex = buildRowtimeIndex;
+ if (buildRowtimeIndex < 0) {
+ throw new IllegalArgumentException("buildRowtimeIndex must be non-negative");
+ }
+ this.generatedJoinCondition = Preconditions.checkNotNull(generatedJoinCondition);
+ this.filterNullKeys = Preconditions.checkNotNull(filterNullKeys);
+ this.loadCompletedTime = Preconditions.checkNotNull(loadCompletedTime);
+ if (this.loadCompletedTime < 0) {
+ throw new IllegalArgumentException("loadCompletedTime must be non-negative");
+ }
+ this.loadCompletedIdleTimeoutMs = loadCompletedIdleTimeoutMs;
+ if (this.loadCompletedIdleTimeoutMs != null && this.loadCompletedIdleTimeoutMs < 0) {
+ throw new IllegalArgumentException("loadCompletedIdleTimeoutMs must be non-negative");
+ }
+ this.minStateTtlMs = stateTtlMs == null ? 0 : stateTtlMs;
+ if (this.minStateTtlMs < 0) {
+ throw new IllegalArgumentException("stateTtlMs must be non-negative");
+ }
+ // maxStateTtlMs is 1.5x of minStateTtlMs
+ this.maxStateTtlMs = this.minStateTtlMs + this.minStateTtlMs / 2;
+ }
+
+ // -------------------------- lifecycle --------------------------
+
+ @Override
+ public boolean useInterruptibleTimers(ReadableConfig config) {
+ return true;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Operator state only — keyed state and timer services are initialized in open()
+ operatorPhaseState =
+ context.getOperatorStateStore()
+ .getUnionListState(
+ new ListStateDescriptor<>(
+ OPERATOR_PHASE_STATE_NAME, StringSerializer.INSTANCE));
+
+ // any LOAD entry → LOAD; empty (fresh start) → LOAD; else JOIN
+ boolean phaseStateExists = false;
+ boolean anyTaskInLoad = false;
+ for (String phase : operatorPhaseState.get()) {
+ phaseStateExists = true;
+ if (Phase.LOAD.name().equals(phase)) {
+ anyTaskInLoad = true;
+ break;
+ }
+ }
+ // we start in LOAD phase if no phaseState exists (no savepoint/checkpoint) or any task was
+ // still in LOAD phase (not all tasks transitioned to JOIN phase).
+ phase = (!phaseStateExists || anyTaskInLoad) ? Phase.LOAD : Phase.JOIN;
+
+ // When restored into JOIN, anchor flipProcTime on the current wall clock so the TTL
+ // handler's post-flip grace window restarts from now.
+ flipProcTime =
+ phase == Phase.JOIN ? getProcessingTimeService().getCurrentProcessingTime() : null;
+
+ currentBuildSideWm = Long.MIN_VALUE;
+ currentProbeSideWm = Long.MIN_VALUE;
+
+ // Initialize metric counters
+ probeBufferedCount = 0L;
+ buildBufferedCount = 0L;
+ maxJoinFanOut = 0L;
+ totalJoinFanOut = 0L;
+ totalProbeJoins = 0L;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ // Setup keyed states
+ buildTableState =
+ getRuntimeContext()
+ .getMapState(
+ new MapStateDescriptor<>(
+ BUILD_TABLE_STATE_NAME, rightType, Types.LONG));
+ buildChangeBuffer =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ BUILD_CHANGE_BUFFER_STATE_NAME, rightType));
+ bufferedAtWmState =
+ getRuntimeContext()
+ .getState(
+ new ValueStateDescriptor<>(BUFFERED_AT_WM_STATE_NAME, Types.LONG));
+ probeBuffer =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType));
+ ttlExpiryState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>(TTL_EXPIRY_STATE_NAME, Types.LONG));
+
+ // Setup timerservice
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Wrap the codegen'd condition with a null-key filter so SQL semantics are honored for
+ // equi-keys whose values may be NULL.
+ final JoinCondition rawCondition =
+ generatedJoinCondition.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ joinCondition = new JoinConditionWithNullFilters(rawCondition, filterNullKeys, this);
+ joinCondition.setRuntimeContext(getRuntimeContext());
+ joinCondition.open(DefaultOpenContext.INSTANCE);
+
+ // Construct null-padded record for left-outer join
+ nullPaddedBuild =
+ isLeftOuterJoin ? new GenericRowData(rightType.toRowType().getFieldCount()) : null;
+
+ // Create output collector
+ collector = new TimestampedCollector<>(output);
+
+ timerService = getInternalTimerService(TIMER_SERVICE_NAME, StringSerializer.INSTANCE, this);
+
+ // Register metrics
+ final MetricGroup metricGroup = getRuntimeContext().getMetricGroup();
+ numStateTtlEvictions = metricGroup.counter(M_NUM_STATE_TTL_EVICTIONS);
+ numUnmatchedProbeRecords = metricGroup.counter(M_NUM_UNMATCHED_PROBE);
+ numUnmatchedBuildRetractions = metricGroup.counter(M_NUM_UNMATCHED_BUILD_RETRACTIONS);
+
+ // Rebuild the buffer counts from restored keyed state so the gauges reflect the
+ // records buffered in this subtask after a restore/rescale.
+ recomputeBufferCountsFromState();
+
+ probeBufferedGauge = new SimpleGauge<>(probeBufferedCount);
+ buildBufferedGauge = new SimpleGauge<>(buildBufferedCount);
+ buildWmGauge = new SimpleGauge<>(currentBuildSideWm);
+ probeWmGauge = new SimpleGauge<>(currentProbeSideWm);
+ maxFanOutGauge = new SimpleGauge<>(maxJoinFanOut);
+ avgFanOutGauge = new SimpleGauge<>(0.0d);
+ metricGroup.gauge(M_NUM_PROBE_BUFFERED, probeBufferedGauge);
+ metricGroup.gauge(M_NUM_BUILD_BUFFERED, buildBufferedGauge);
+ metricGroup.gauge(M_CURRENT_BUILD_WM, buildWmGauge);
+ metricGroup.gauge(M_CURRENT_PROBE_WM, probeWmGauge);
+ metricGroup.gauge(M_MAX_JOIN_FAN_OUT, maxFanOutGauge);
+ metricGroup.gauge(M_AVG_JOIN_FAN_OUT, avgFanOutGauge);
+ phaseGauge = () -> phase == null ? -1 : phase.ordinal();
+ metricGroup.gauge(M_CURRENT_PHASE, phaseGauge);
+
+ // Mark the build-side input (index 1) as permanently idle in the inherited
+ // combinedWatermark accounting. This operator never forwards build-side WMs nor
+ // build-side idle status: it absorbs both.
+ combinedWatermark.updateStatus(1, true);
+
+ // Register the load-completed idle-timeout timer if it is configured.
+ if (phase == Phase.LOAD && loadCompletedIdleTimeoutMs != null) {
+ scheduleIdleFlipTimer();
+ }
+
+ LOG.info(
+ "Opened LateralSnapshotJoinOperator: phase={}, leftOuter={}, loadCompletedTime={}, "
+ + "idleTimeoutMs={}, stateTtlMs=[{}, {}], buildRowtimeIndex={}, "
+ + "restoredProbeBuffered={}, restoredBuildBuffered={}",
+ phase,
+ isLeftOuterJoin,
+ loadCompletedTime,
+ loadCompletedIdleTimeoutMs,
+ minStateTtlMs,
+ maxStateTtlMs,
+ buildRowtimeIndex,
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ /**
+ * Recomputes the in-memory buffer counter ({@link #probeBufferedCount}, {@link
+ * #buildBufferedCount}) for the corresponding metrics by scanning the restored keyed buffer
+ * state.
+ */
+ private void recomputeBufferCountsFromState() throws Exception {
+ final KeyedStateBackend<Object> backend = getKeyedStateBackend();
+
+ LOG.info("Start recomputing buffered-record metrics from restored keyed state...");
+
+ probeBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(PROBE_BUFFER_STATE_NAME, leftType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ probeBufferedCount++;
+ }
+ });
+
+ buildBufferedCount = 0L;
+ backend.applyToAllKeys(
+ VoidNamespace.INSTANCE,
+ VoidNamespaceSerializer.INSTANCE,
+ new ListStateDescriptor<>(BUILD_CHANGE_BUFFER_STATE_NAME, rightType),
+ (key, state) -> {
+ for (RowData ignored : state.get()) {
+ buildBufferedCount++;
+ }
+ });
+
+ LOG.info(
+ "Finished recomputing buffered-record metrics from restored keyed state: "
+ + "probeBuffered={}, buildBuffered={}",
+ probeBufferedCount,
+ buildBufferedCount);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+ operatorPhaseState.update(List.of(phase.name()));
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ if (joinCondition != null) {
+ joinCondition.close();
+ }
+ super.close();
+ }
+
+ // -------------------------- input row processing --------------------------
+
+ @Override
+ public void processElement1(StreamRecord<RowData> element) throws Exception {
+ RowData probe = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ applyBufferedChangesIfReady();
+ if (phase == Phase.LOAD) {
+ probeBuffer.add(probe);
+ timerService.registerEventTimeTimer(NS_FLIP, FLIP_JOIN_TIMER_TS);
+ probeBufferedGauge.update(++probeBufferedCount);
+ } else {
+ joinProbeRow(probe);
+ }
+ refreshStateTtl();
+ }
+
+ @Override
+ public void processElement2(StreamRecord<RowData> element) throws Exception {
+ RowData build = element.getValue();
+ // Apply any buffered build-side changes if the build-side watermark has advanced.
+ Long bufferedAt = applyBufferedChangesIfReady();
+ // Buffer the new change, tagged with the current build-side watermark.
+ buildChangeBuffer.add(build);
+ if (bufferedAt == null || bufferedAt != currentBuildSideWm) {
+ bufferedAtWmState.update(currentBuildSideWm);
+ }
+ buildBufferedGauge.update(++buildBufferedCount);
+ refreshStateTtl();
+ }
+
+ // -------------------------- watermark processing --------------------------
+
+ @Override
+ public void processWatermark1(Watermark mark) throws Exception {
+ // Probe-side watermark: held back during LOAD (neither advances the timer service nor is
+ // forwarded), forwarded during JOIN.
+ if (phase == Phase.JOIN) {
+ super.processWatermark1(mark);
+ }
+ currentProbeSideWm = Math.max(currentProbeSideWm, mark.getTimestamp());
+ probeWmGauge.update(currentProbeSideWm);
+ }
+
+ @Override
+ public void processWatermark2(Watermark mark) throws Exception {
+ // Build-side watermark: NEVER forwarded; never advances the timer service.
+ long ts = mark.getTimestamp();
+ currentBuildSideWm = Math.max(currentBuildSideWm, ts);
+ buildWmGauge.update(currentBuildSideWm);
+ if (phase == Phase.LOAD) {
+ if (currentBuildSideWm >= loadCompletedTime) {
+ // we reached the flip point. Transition to JOIN phase.
+ transitionToJoinPhase(FlipTrigger.BUILD_WATERMARK);
+ } else if (loadCompletedIdleTimeoutMs != null) {
+ // we got a new build-side wm. Reschedule the idle timer (if it was configured)
+ rescheduleIdleFlipTimer();
+ }
+ }
+ }
+
+ @Override
+ protected void processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
+ throws Exception {
+ if (index == 1) {
+ // Build-side idle status is absorbed entirely. partial[1] is initialized idle in
+ // open() and stays that way regardless of source-side toggles, so combined accounting
+ // is always driven by the probe side alone.
+ return;
+ }
+ if (phase == Phase.LOAD) {
+ // During LOAD, nothing is emitted downstream — neither watermarks nor status. But we
+ // do track partial[0]'s idle bit so that, after the flip, the operator has an accurate
+ // view of the probe-side's idle state.
+ combinedWatermark.updateStatus(0, watermarkStatus.isIdle());
+ return;
+ }
+ super.processWatermarkStatus(watermarkStatus, index);
+ }
+
+ // -------------------------- timers processing --------------------------
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, String> timer) throws Exception {
+ String ns = timer.getNamespace();
+ // the NS_FLIP timers are fired when the operator transitions from LOAD to JOIN phase.
+ if (NS_FLIP.equals(ns)) {
+ // If a recovery happened before, there might be buffered build-side changes.
+ // Apply them before joining the buffered probe-side records.
+ applyBufferedChanges();
+ // Join each buffered probe row.
+ long drained = 0;
+ for (RowData p : probeBuffer.get()) {
+ joinProbeRow(p);
+ drained++;
+ }
+ probeBuffer.clear();
+ probeBufferedCount = Math.max(0, probeBufferedCount - drained);
+ probeBufferedGauge.update(probeBufferedCount);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, String> timer) throws Exception {
+ // TTL timers run on processing time so semantics match Flink's standard StateTtlConfig.
+ if (!NS_TTL.equals(timer.getNamespace())) {
+ return;
+ }
+ if (minStateTtlMs == 0) {
+ // TTL wasn't configured and shouldn't have registered any timers.
+ return;
+ }
+ Long deadline = ttlExpiryState.value();
+ if (deadline == null || timer.getTimestamp() != deadline) {
+ return; // stale timer fire
+ }
+ long now = getProcessingTimeService().getCurrentProcessingTime();
+ // check if we need to reschedule the ttl timer. This is necessary if
+ // a) we're still in LOAD phase, or
+ // b) if we're in JOIN but the flip happened less than stateTtlMs ago.
+ if (phase == Phase.LOAD || (flipProcTime != null && now < flipProcTime + minStateTtlMs)) {
+ // set the new TTL timer maxStateTtlMs ahead
+ long newDeadline =
+ phase == Phase.LOAD ? now + maxStateTtlMs : flipProcTime + maxStateTtlMs;
+ timerService.registerProcessingTimeTimer(NS_TTL, newDeadline);
+ ttlExpiryState.update(newDeadline);
+ return;
+ }
+ // clear all per-key state
+ buildTableState.clear();
+ // Discount any build-side changes still buffered for this key before clearing.
+ long evictedBuffered = 0;
+ for (RowData ignored : buildChangeBuffer.get()) {
+ evictedBuffered++;
+ }
+ buildBufferedCount = Math.max(0, buildBufferedCount - evictedBuffered);
+ buildBufferedGauge.update(buildBufferedCount);
+ buildChangeBuffer.clear();
+ bufferedAtWmState.clear();
+ ttlExpiryState.clear();
+ // probeBuffer should be empty because we are in JOIN phase, but clear out just in case.
+ // hence, we're also not updating the probeBufferedGauge.
+ probeBuffer.clear();
+ numStateTtlEvictions.inc();
+ }
+
+ /**
+ * Registers the load-completion idle-timeout timer. No-op when the timeout is not configured.
+ */
+ private void scheduleIdleFlipTimer() {
+ if (loadCompletedIdleTimeoutMs == null) {
+ return;
+ }
+ long deadline =
+ getProcessingTimeService().getCurrentProcessingTime() + loadCompletedIdleTimeoutMs;
+ idleFlipTimer =
+ getProcessingTimeService()
+ .registerTimer(
+ deadline, t -> transitionToJoinPhase(FlipTrigger.IDLE_TIMEOUT));
+ }
+
+ /** Updates the idle flip timer. */
+ private void rescheduleIdleFlipTimer() {
+ cancelIdleFlipTimer();
+ scheduleIdleFlipTimer();
+ }
+
+ /** Deactivates the currently registered idle flip timer. */
+ private void cancelIdleFlipTimer() {
+ if (idleFlipTimer != null) {
+ idleFlipTimer.cancel(false);
+ idleFlipTimer = null;
+ }
+ }
+
+ // -------------------------- core logic --------------------------
+
+ /**
+ * Transition from LOAD to JOIN.
+ *
+ * <p><b>Invocation context</b>: This method runs in a NON-KEYED context. The two callers are
+ * (a) {@link #processWatermark2}, which is invoked by the framework without a key context, and
+ * (b) {@link #idleFlipTimer}, which fires from the operator-level processing-time service.
+ * Therefore {@code flipToJoinPhase()} itself must not access keyed state. Per-key work (the
+ * buffered probe flush) is delegated to {@link #onEventTime} via {@code timeServiceManager
+ * .advanceWatermark(...)} below — that path establishes the correct key context for each fired
+ * timer before invoking the callback.
+ */
+ private void transitionToJoinPhase(FlipTrigger trigger) throws Exception {
+ if (phase == Phase.JOIN) {
+ return;
+ }
+ LOG.info(
+ "Flipping phase LOAD -> JOIN (trigger={}): buildWm={}, loadCompletedTime={}. "
+ + "Joining buffered probe records...",
+ trigger,
+ currentBuildSideWm,
+ loadCompletedTime);
+ phase = Phase.JOIN;
+ // Record the flip wall-clock so the TTL handler can grant a grace period of
+ // stateTtlMs after the flip before any build-only key becomes eligible for eviction.
+ // Without this anchor, keys loaded long before the flip would be evicted as soon as the
+ // first TTL fire after the flip happens.
+ flipProcTime = getProcessingTimeService().getCurrentProcessingTime();
+ // disable idle flip timer
+ cancelIdleFlipTimer();
+ // Fire all per-key flip timers (TS=1) so any probes buffered during LOAD are joined.
+ long advanceTo = Math.max(currentProbeSideWm, FLIP_JOIN_TIMER_TS);
+ if (timeServiceManager != null) {
+ timeServiceManager.advanceWatermark(new Watermark(advanceTo));
+ }
+ // Track the idle status which is cleared by updateWatermark()
+ boolean probeIdleAtFlip = combinedWatermark.isIdle();
+ // Emit the last observed probe-side wm downstream
+ if (currentProbeSideWm != Long.MIN_VALUE) {
+ combinedWatermark.updateWatermark(0, currentProbeSideWm);
+ output.emitWatermark(new Watermark(currentProbeSideWm));
+ // reset the idle status
+ combinedWatermark.updateStatus(0, probeIdleAtFlip);
+ }
+ // If the probe-side was idle at flip time, propagate the idle status downstream
+ if (probeIdleAtFlip) {
+ output.emitWatermarkStatus(WatermarkStatus.IDLE);
+ }
+ LOG.info(
+ "Completed flip to JOIN phase (trigger={}): buffered probe records joined and "
+ + "drained, emittedProbeWm={}",
+ trigger,
+ currentProbeSideWm == Long.MIN_VALUE ? "none" : currentProbeSideWm);
+ }
+
+ /**
+ * Joins a probe-side row against the current build-side table and applies the join predicate.
+ * Returns a null-padded result if the row doesn't match any build-side row and this is a LEFT
+ * OUTER join.
+ */
+ private void joinProbeRow(RowData probe) throws Exception {
+ boolean matched = false;
+ // Number of result rows emitted for this probe row (the join fan-out).
+ long fanOut = 0;
+ for (Map.Entry<RowData, Long> entry : buildTableState.entries()) {
+ RowData buildRow = entry.getKey();
+ long count = entry.getValue();
+ if (joinCondition.apply(probe, buildRow)) {
+ matched = true;
+ // Each emitted record uses a fresh JoinedRowData wrapper.
+ // Reusing a row object here is unsafe when subsequent collects mutate it.
+ for (long i = 0; i < count; i++) {
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, buildRow);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ }
+ }
+ if (!matched && isLeftOuterJoin) {
+ // No join match, emit a null-padded LEFT OUTER join result
+ JoinedRowData out = new JoinedRowData();
+ out.replace(probe, nullPaddedBuild);
+ out.setRowKind(RowKind.INSERT);
+ collector.collect(out);
+ fanOut++;
+ }
+ if (!matched && !isLeftOuterJoin) {
+ numUnmatchedProbeRecords.inc();
+ }
+ // Update fan-out statistics (fan-out = number of result rows emitted for this probe).
+ totalProbeJoins++;
+ totalJoinFanOut += fanOut;
+ if (fanOut > maxJoinFanOut) {
+ maxJoinFanOut = fanOut;
+ maxFanOutGauge.update(maxJoinFanOut);
+ }
+ avgFanOutGauge.update(((double) totalJoinFanOut) / totalProbeJoins);
+ }
+
+ /**
+ * Applies the buffered build-side changes if the build-side watermark advanced since last
+ * buffer application. This ensures that we apply buffered changes atomically once their
+ * corresponding build-side WM is passed.
+ *
+ * @return the watermark tag of the buffer that is still in effect for this key after this call,
+ * or {@code null} if nothing is buffered.
+ */
+ @Nullable
+ private Long applyBufferedChangesIfReady() throws Exception {
+ Long bufferedAt = bufferedAtWmState.value();
+ if (bufferedAt == null) {
+ // Nothing buffered for this key.
+ return null;
+ }
+ if (currentBuildSideWm > bufferedAt) {
+ // the build-side wm advanced. Buffered changes can be applied atomically now.
+ applyBufferedChanges();
+ return null;
+ } else if (phase == Phase.JOIN && currentBuildSideWm == Long.MIN_VALUE) {
+ // JOIN-phase fallback: no build-side watermark has ever been observed by this subtask.
+ // This happens
+ // (a) after a recovery (e.g. JOIN-phase rescaled into this subtask), or
+ // (b) when the operator flipped via the idle-timeout fallback and no build-side
+ // watermark arrived afterwards.
+ // In either case we apply the buffered changes now because we do not know when the next
+ // build-side WM will arrive (if it ever will).
+ applyBufferedChanges();
+ return null;
+ }
+ return bufferedAt;
+ }
+
+ /**
+ * Apply all buffered build-side changes for the current key to {@code buildTableState} in
+ * event-time order, then clear the buffer state.
+ */
+ private void applyBufferedChanges() throws Exception {
+ List<RowData> changes = new ArrayList<>();
+ for (RowData c : buildChangeBuffer.get()) {
+ changes.add(c);
+ }
+ // Apply in event-time order; read the row-time before applyBuildChange mutates the kind.
+ changes.sort(this::compareBuildChanges);
+ for (RowData c : changes) {
+ applyBuildChange(c);
Review Comment:
In the current implementation we gate the application of the buffer with `currentBuildSideWm > bufferedAt`.
`bufferedAt` is the build-side WM at the time when the first record was put into the buffer and `currentBuildSideWm` is the latest observed build-side WM.
This mechanism essentially resembles the common processing of data by watermarks to ensure completeness.
However, this operator works a bit differently, because we only want to take the build-side WMs into account (and not the combined WM of probe and build side). The reason is that we don't want the build-side WM to affect the output WM, so we cannot use the regular timer mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org