You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pvary (via GitHub)" <gi...@apache.org> on 2024/06/11 14:46:18 UTC
[PR] Flink: Maintenance - TriggerManager [iceberg]
pvary opened a new pull request, #10484:
URL: https://github.com/apache/iceberg/pull/10484
The responsibility of the Trigger Manager is to start the Maintenance Tasks based on the incoming Table Change messages and prevent overlapping Maintenance Task runs. The event time of the Trigger messages should be monotonically increasing, as this will be used as a watermark to separate Maintenance Task runs.
Implementation for https://github.com/apache/iceberg/issues/10301
This will be used for Maintenance Trigger Manager as described in the [design doc](https://docs.google.com/document/d/16g3vR18mVBy8jbFaLjf2JwAANuYOmIwr15yDDxovdnA). The Trigger Manager paragraph describes the requirements in more detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699876545
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
Review Comment:
That's a good point. Added the check to the `init` `shouldRestoreTasks` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701465612
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
Review Comment:
Added:
```
// At this point the output contains the recovery trigger and the real trigger
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1713959444
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ @VisibleForTesting static final String RUNNING_TAG_PREFIX = "__flink_maintenance_running_";
+ @VisibleForTesting static final String RECOVERING_TAG_PREFIX = "__flink_maintenance_recovering_";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG_PREFIX);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG_PREFIX);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockPrefix;
+
+ public Lock(Table table, String lockPrefix) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockPrefix, "Lock key should not be null");
+ this.table = table;
+ this.lockPrefix = lockPrefix;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held. The relevant tag: {}", findLock());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ String lockKey = lockPrefix + UUID.randomUUID();
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ unused -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.createTag(lockKey, table.currentSnapshot().snapshotId());
+ manage.commit();
+ LOG.debug("Lock created");
+ } catch (Exception e) {
+ if (!isHeld()) {
+ LOG.warn("Retrying lock after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock created, hiding exception", e);
+ }
+ }
+ });
+ } catch (Exception e) {
+ LOG.info("Concurrent lock created. Is there a concurrent maintenance job running?", e);
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ public void unlock() {
+ table.refresh();
+
+ Optional<String> lockKey = findLock();
+ if (lockKey.isPresent()) {
+ Tasks.foreach(lockKey.get())
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ key -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.removeTag(key);
+ manage.commit();
+ } catch (Exception e) {
+ table.refresh();
+ if (table.refs().containsKey(key)) {
+ LOG.warn("Retrying lock removal after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock removed, hiding exception", e);
+ }
+ }
+ });
+ LOG.debug("Lock removed");
+ } else {
+ LOG.warn(
+ "Missing lock, can not remove. Found only the following tags: {}.",
+ table.refs().keySet());
+ }
+ }
+
+ @Override
+ public boolean isHeld() {
+ table.refresh();
+ return findLock().isPresent();
+ }
+
+ private Optional<String> findLock() {
+ List<String> locks =
+ table.refs().keySet().stream()
+ .filter(tag -> tag.startsWith(lockPrefix))
+ .collect(Collectors.toList());
+ Preconditions.checkArgument(locks.size() < 2, "Invalid lock state: %s", locks);
Review Comment:
Removed `TagBasedLockFactory`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724717076
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724719157
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockFactoryBase.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+abstract class TestLockFactoryBase {
+ protected TriggerLockFactory lockFactory;
+
+ abstract TriggerLockFactory lockFactory();
+
+ @BeforeEach
+ void before() {
+ this.lockFactory = lockFactory();
+ lockFactory.open();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testTryLock() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock1.tryLock()).isFalse();
+ assertThat(lock2.tryLock()).isFalse();
+ }
+
+ @Test
+ void testUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ @Test
+ void testNoConflict() {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1726595300
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+
+ @VisibleForTesting
+ static final String INIT_LOCK_TABLES_PROPERTY = "flink-maintenance.lock.jdbc.init-lock-tables";
+
+ private static final String LOCK_TABLE_NAME = "flink_maintenance_lock";
+ private static final int LOCK_ID_MAX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ String.format(
+ "CREATE TABLE %s "
+ + "(LOCK_TYPE CHAR(1) NOT NULL, "
+ + "LOCK_ID VARCHAR(%s) NOT NULL, "
+ + "INSTANCE_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, LOCK_ID))",
+ LOCK_TABLE_NAME, LOCK_ID_MAX_LENGTH);
+
+ private static final String CREATE_LOCK_SQL =
+ String.format(
+ "INSERT INTO %s (LOCK_TYPE, LOCK_ID, INSTANCE_ID) VALUES (?, ?, ?)", LOCK_TABLE_NAME);
+ private static final String GET_LOCK_SQL =
+ String.format("SELECT INSTANCE_ID FROM %s WHERE LOCK_TYPE=? AND LOCK_ID=?", LOCK_TABLE_NAME);
+ private static final String DELETE_LOCK_SQL =
+ String.format(
+ "DELETE FROM %s WHERE LOCK_TYPE=? AND LOCK_ID=? AND INSTANCE_ID=?", LOCK_TABLE_NAME);
+
+ private final String uri;
+ private final String lockId;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ /**
+ * Creates a new {@link TriggerLockFactory}. The issuer should be unique between the users of the
Review Comment:
good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639281217
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698470388
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
Review Comment:
Added close to the LockManagerFactory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698469479
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698852703
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(Table table, String lockKey) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockKey, "Lock key should not be null");
+ this.table = table;
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ unused -> {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.createTag(lockKey, table.currentSnapshot().snapshotId());
+ manage.commit();
+ LOG.debug("Lock created");
+ });
+ } catch (Exception e) {
+ LOG.info("Concurrent lock created. Stop concurrent maintenance jobs.", e);
Review Comment:
nit: this error msg doesn't seem correct. it just failed to acquire lock. it doesn't actually `stop` anything.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(Table table, String lockKey) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockKey, "Lock key should not be null");
+ this.table = table;
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
Review Comment:
wondering if retry is safe here. Could the lock/unblock action initiated by multiple threads/operator subtasks?
here is one scenario I am thinking about.
- t1: operator 1 calls unlock, it actually removed the tag by the catalog service/backend, but response failed to get back due to network issue
- t2: operator 2 calls lock, and created the tag successfully.
- t3: operator 1 retry the unlock and removed the tag
If the above scenario couldn't happen, let's call out the constraints in the Javadoc of this class.
also if the tag is already removed, there is no need to retry. but I guess this is not a correctness problem. just minor behavior improvement. don't know if we can easily distinguish this case with other errors.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService) throws Exception {
+ if (!inited) {
+ long current = timerService.currentProcessingTime();
+
+ // Initialize from state
+ this.nextEvaluationTime = nextEvaluationTimeState.value();
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ if (shouldRestoreTasks) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a cleanup lock
Review Comment:
nit: for consistency, maybe `acquire the recovery lock`
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
Review Comment:
let's call out its intended scope/limitation of the tag based locking: mainly only working with single job scenario. while we are clarifying that, maybe also call out multiple Flink jobs running compaction is not recommended.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
Review Comment:
I saw 4 places where `schedule()` were called in `TriggerManager`
1. init: `if (shouldRestoreTasks)`
2. checkAndFire: `if (recoveryLock.isHeld())`
3. checkAndFire: `if (lock.tryLock())` and `else` (these two are mutually exclusive)
1 and 2 can co-exist. 1 and 3 also seems to can happen together.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
Review Comment:
oh. I didn't mean to "run the checkAndFire when the nextTime is larger then the current". anyway, I guess it is not important in this else section.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(Table table, String lockKey) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockKey, "Lock key should not be null");
+ this.table = table;
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
Review Comment:
also how do we deal with orphaned locks where unlock failed (e.g. due to network issue with catalog service)
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
Review Comment:
what other operator (besides TriggerManager here) subtask can acquire lock?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService) throws Exception {
+ if (!inited) {
+ long current = timerService.currentProcessingTime();
+
+ // Initialize from state
+ this.nextEvaluationTime = nextEvaluationTimeState.value();
Review Comment:
it is probably cleaner to do line 296-306 in `initializeState` method in the `if (context.isRestored())` block
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ table.refresh();
+ Map<String, SnapshotRef> refs = table.refs();
+ if (isHeld()) {
+ LOG.info("Lock is already held by someone: {}", refs.keySet());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
Review Comment:
alright fair enough
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
Review Comment:
I was saying perform the mod operation here to make sure `startsFrom` is always a valid position. This way, if we log this value, it will show correct value.
```
startsFrom = (taskToStart + 1) % evaluators.size();
```
instead of in `nextTrigger` method
```
int normalizedStartingPos = startPos % evaluators.size();
```
This should affect fairness, right?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
Review Comment:
nit: for consistency, should it be `recover lock` in the log msg?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
Review Comment:
`nextTime` is only used in the `else` case without running `checkAndFire`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701399704
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}. The current
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701447388
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTagBasedLockFactory.java:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TagBasedLockFactory.RUNNING_TAG_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.CommitFailedException;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTagBasedLockFactory extends TestLockFactoryBase {
Review Comment:
`TableLoader` is needed for the `TagBasedLockFactory`. Also SQL is used to create the table, and add a snapshot to it. There could be other ways to get the desired setup, but I thought that reusing the `OperatorTestBase` add the least mental complexity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1726304565
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
+ return conn.prepareStatement(CREATE_LOCK_TABLE_SQL).execute();
Review Comment:
my mistake. I thought the boolean return value from `execute` is success or failure. which it is not. So we can ignore this comment.
```
true if the first result is a ResultSet object; false if the first result is an update count or there is no result
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1713948194
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
Review Comment:
Could you please describe the situation where this could happen?
I still don't see it.
1. is fixed with the check I have added to the schedule.
2. and 3. `checkAndFire` is called in 2 places:
```
public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
throws Exception {
[..]
if (nextEvaluationTime == null) {
checkAndFire(current, ctx.timerService(), out);
} else {
```
and
```
@Override
public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
init(out, ctx.timerService());
this.nextEvaluationTime = null;
checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
}
```
I think in both cases the `nextEvaluationTime` is definitely `null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698414530
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#issuecomment-2304799537
Merged to master.
Thanks for the review @stevenzwu, @singhpk234 and @gyfora!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1642821801
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
Review Comment:
So the issue is with the timers:
```
java.lang.UnsupportedOperationException: Setting timers is only supported on a keyed streams.
```
The method is there, but we can not set the timer from a non-keyed `ProcessFunction`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1697274018
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ table.refresh();
+ Map<String, SnapshotRef> refs = table.refs();
+ if (isHeld()) {
+ LOG.info("Lock is already held by someone: {}", refs.keySet());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
Review Comment:
I think it is much easier to follow the logic around the locking if we don't complicate it event more.
For example we should rethink the logic around the recovery lock to be sure that the cleanup logic is working if there is no commits in the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1707718356
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerLockFactory.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.Closeable;
+import java.io.Serializable;
+import org.apache.flink.annotation.Experimental;
+
+/** Lock interface for handling locks for the Flink Table Maintenance jobs. */
+@Experimental
+public interface TriggerLockFactory extends Serializable, Closeable {
+ void open();
+
+ Lock createLock();
+
+ Lock createRecoveryLock();
+
+ interface Lock {
+ /**
+ * Tries to acquire a lock with a given key. Anyone already holding a lock would prevent
+ * acquiring this lock. Not reentrant.
+ *
+ * <p>Called by {@link TriggerManager}. Implementations could assume that are no concurrent
+ * calls for this method.
+ *
+ * @return <code>true</code> if the lock is acquired by this job, <code>false</code> if the lock
+ * is already held by someone
+ */
+ boolean tryLock();
+
+ /**
+ * Checks if the lock is already taken.
+ *
+ * @return <code>true</code> if the lock is held by someone
+ */
+ boolean isHeld();
+
+ // TODO: Fix the link to the LockRemover when we have a final name and implementation
+ /**
+ * Releases the lock. Should not fail if the lock is not held by anyone.
+ *
+ * <p>Called by LockRemover. Implementations could assume that are no concurrent calls for this
+ * method.
+ */
+ void unlock();
+ }
Review Comment:
Notice the different requirements for the methods. This lock is not reentrant, and the unlock removes locks held by others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724521388
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
Review Comment:
Used the string concatenation since that was used by the JdbcCatalog in the `JdbcUtil`, but I also prefer the format, so I have changed the way to handle sql statements for the `JdbcLockFactory`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1636077522
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/Trigger.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+@Internal
+class Trigger {
+ private final long timestamp;
+ private final SerializableTable table;
+ private final Integer taskId;
+ private final boolean isCleanUp;
+
+ private Trigger(long timestamp, SerializableTable table, Integer taskId, boolean isCleanUp) {
+ this.timestamp = timestamp;
+ this.table = table;
+ this.taskId = taskId;
Review Comment:
I'm undecided between 2 solutions:
1. The trigger contains the ID of the Maintenance Task (implemented in this PR)
2. Using side outputs to separate out the triggers for the Maintenance Tasks
With the 1st approach the flow looks like this:
- TriggerManager (1 instance)
- Watermark generator (1 instance)
- Splitter to start the specific tasks (1 instance)
- Maintenance Tasks (n instance)
With the 2nd approach the flow looks like this:
- TriggerManager (1 instance)
- Watermark generator (n instance)
- Maintenance Tasks (n instance)
IMHO the 1st is somewhat easier to read, the 2nd has fewer different type of operators.
I don't really have a clear preference?
Do you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1635591884
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MetricConstants.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+public class MetricConstants {
Review Comment:
nit: `TableMaintenanceMetrics`?
##########
core/src/main/java/org/apache/iceberg/SerializableTable.java:
##########
@@ -397,6 +398,25 @@ private String errorMsg(String operation) {
return String.format("Operation %s is not supported after the table is serialized", operation);
}
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ SerializableTable other = (SerializableTable) o;
+ return Objects.equals(metadataFileLocation, other.metadataFileLocation);
+ }
+
+ @Override
+ public int hashCode() {
+ return metadataFileLocation.hashCode();
Review Comment:
Can we avoid the requirement of `equals` and `hashCode`?
Note that `metadataFileLocation` can be null. There were also discussion on potentially removing the metadata json file requirement from the spec.
```
private String metadataFileLocation(Table table) {
if (table instanceof HasTableOperations) {
TableOperations ops = ((HasTableOperations) table).operations();
return ops.current().metadataFileLocation();
} else if (table instanceof BaseMetadataTable) {
return ((BaseMetadataTable) table).table().operations().current().metadataFileLocation();
} else {
return null;
}
}
```
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
Review Comment:
why is this a `KeyedProcessFunction` after `keyBy`? I thought it will always be parallelism of 1.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
Review Comment:
do we need `List<TableChange>` here? `TableChange` is mergeable. a single instance would be enough?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerEvaluator.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.List;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Internal
+class TriggerEvaluator implements Serializable {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerEvaluator.class);
+ private final List<Predicate> predicates;
+
+ private TriggerEvaluator(List<Predicate> predicates) {
+ Preconditions.checkArgument(!predicates.isEmpty(), "Provide at least 1 condition.");
+
+ this.predicates = predicates;
+ }
+
+ boolean check(TableChange event, long lastTime, long currentTime) {
Review Comment:
nit: may indicate where it is milli or micro in the arg names
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
It is probably best not to use tag for this purpose. `Tag` is a named snapshot id. doesn't seem to be the right fit.
Alternative would be table property. I would love to hear others' inputs on the lock implementation.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/Trigger.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+@Internal
+class Trigger {
+ private final long timestamp;
+ private final SerializableTable table;
+ private final Integer taskId;
+ private final boolean isCleanUp;
+
+ private Trigger(long timestamp, SerializableTable table, Integer taskId, boolean isCleanUp) {
+ this.timestamp = timestamp;
+ this.table = table;
+ this.taskId = taskId;
Review Comment:
what is taskId for? it doesn't seem to be exposed to outside.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
Review Comment:
nit: `init` name is confusing. it is not initialization of this function/operator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701369847
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.UUID;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ @VisibleForTesting static final String RUNNING_TAG_PREFIX = "__flink_maintenance_running_";
+ @VisibleForTesting static final String RECOVERING_TAG_PREFIX = "__flink_maintenance_recovering_";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG_PREFIX);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG_PREFIX);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockPrefix;
+
+ public Lock(Table table, String lockPrefix) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockPrefix, "Lock key should not be null");
+ this.table = table;
+ this.lockPrefix = lockPrefix;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held. The relevant tag: {}", findLock());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ String lockKey = lockPrefix + UUID.randomUUID();
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ unused -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.createTag(lockKey, table.currentSnapshot().snapshotId());
+ manage.commit();
+ LOG.debug("Lock created");
+ } catch (Exception e) {
+ if (!isHeld()) {
+ LOG.warn("Retrying lock after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock created, hiding exception", e);
+ }
+ }
+ });
+ } catch (Exception e) {
+ LOG.info("Concurrent lock created. Is there a concurrent maintenance job running?", e);
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ public void unlock() {
+ table.refresh();
+
+ Optional<String> lockKey = findLock();
+ if (lockKey.isPresent()) {
+ Tasks.foreach(lockKey.get())
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ key -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.removeTag(key);
+ manage.commit();
+ } catch (Exception e) {
+ table.refresh();
+ if (table.refs().containsKey(key)) {
+ LOG.warn("Retrying lock removal after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock removed, hiding exception", e);
+ }
+ }
+ });
+ LOG.debug("Lock removed");
+ } else {
+ LOG.warn(
+ "Missing lock, can not remove. Found only the following tags: {}.",
+ table.refs().keySet());
+ }
+ }
+
+ @Override
+ public boolean isHeld() {
+ table.refresh();
+ return findLock().isPresent();
+ }
+
+ private Optional<String> findLock() {
+ return table.refs().keySet().stream().filter(tag -> tag.startsWith(lockPrefix)).findAny();
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701460602
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
Review Comment:
Yes.
This test is for testing the `TriggerEvaluator` timeout handling.
We need a known time for a successful trigger (`newTime`) for a previous trigger, and later we can check if the `TriggerEvaluator.timeout` is handled correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1635930764
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
The only requirement is `atomicity`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1643986681
##########
core/src/main/java/org/apache/iceberg/SerializableTable.java:
##########
@@ -397,6 +398,25 @@ private String errorMsg(String operation) {
return String.format("Operation %s is not supported after the table is serialized", operation);
}
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ SerializableTable other = (SerializableTable) o;
+ return Objects.equals(metadataFileLocation, other.metadataFileLocation);
+ }
+
+ @Override
+ public int hashCode() {
+ return metadataFileLocation.hashCode();
Review Comment:
Removed, added a utility method for comparison as it is used only in the tests ATM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639276069
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
Review Comment:
We don't support this scenario ATM.
We can make it work with named locks, where we can identify the owner of the currently held lock by some means, but then we need to solve the prioritizing and starving of these concurrent Maintenance Jobs. I decided against solving that issue ATM.
And this is the feature where we will need the `cleanup` config again to remove all of the stale locks for the given table... 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705638405
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/MetricsReporterFactoryForTests.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.MetricReporterFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+
+public class MetricsReporterFactoryForTests implements MetricReporterFactory {
Review Comment:
How can I access the metrics from my test to check the values of the relevant counters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701945987
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
Review Comment:
Separated out the `triggered` variable to a boolean. So it is easier to understand the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701471050
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
Review Comment:
Added:
```
// Simulate the action of the recovered maintenance task lock removal when it finishes
```
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699877237
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
Review Comment:
I leave this open, so you can double check the fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1700304518
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(Table table, String lockKey) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockKey, "Lock key should not be null");
+ this.table = table;
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
Review Comment:
> > wondering if retry is safe here. Could the lock/unblock action initiated by multiple threads/operator subtasks?
>
> This is a valid issue. We need to write a more complicated locking with lockIds, and checks like we have in `MetastoreLock` and `LockRequest`. The main idea is to have something like `__flink_maitenance_<RND>` for the lock.
>
> * For checking the lock, we check if there is any tag starting with `__flink_maitenance_`
> * For creating the lock, we create a tag with a `__flink_maitenance_<RND>`
> * For retries for creation, we check if the lock with `RND` is created, and if not, try to create again
> * For removal, we list the locks and remove the tag with `__flink_maitenance_<RND>`
> * For retries for removal, we check if the lock with the `RND` is removed, and if not, try to remove again
Fixed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724522494
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
Review Comment:
Done
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724696254
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
Review Comment:
Added comment to the `unlock` and even refactored the code to cover more situations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1697286779
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
Review Comment:
If we trigger the `checkAndFire` when the `nextTime` is larger then the `current`, then we will have a trigger by this new `checkAndFire`, and also the timer could be called to. It is better to avoid this situation altogether by relying on the timer to trigger when it is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1726213077
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockFactoryBase.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+abstract class TestLockFactoryBase {
+ protected TriggerLockFactory lockFactory;
+
+ abstract TriggerLockFactory lockFactory();
+
+ @BeforeEach
+ void before() {
+ this.lockFactory = lockFactory();
+ lockFactory.open();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testTryLock() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock1.tryLock()).isFalse();
+ assertThat(lock2.tryLock()).isFalse();
+ }
+
+ @Test
+ void testUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ @Test
+ void testNoConflict() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createRecoveryLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock2.tryLock()).isTrue();
+ }
+
+ @Test
+ void testDoubleUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ lock.unlock();
Review Comment:
unlock can return false when the lock doesn't exist. then here we can assert that 1st unlock should return true and 2nd unlock should return false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724524205
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724718482
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
Review Comment:
Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724721975
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockFactoryBase.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+abstract class TestLockFactoryBase {
+ protected TriggerLockFactory lockFactory;
+
+ abstract TriggerLockFactory lockFactory();
+
+ @BeforeEach
+ void before() {
+ this.lockFactory = lockFactory();
+ lockFactory.open();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testTryLock() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock1.tryLock()).isFalse();
+ assertThat(lock2.tryLock()).isFalse();
+ }
+
+ @Test
+ void testUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ @Test
+ void testNoConflict() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createRecoveryLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock2.tryLock()).isTrue();
+ }
+
+ @Test
+ void testDoubleUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ lock.unlock();
Review Comment:
The return value would not be used anywhere, and I prefer the leanest API as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701495274
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
Review Comment:
I'm not able to understand this question 😢
Could you please elaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701501963
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
Review Comment:
This test is only for the metrics.
The functionality is tested by `testMinFireDelay`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701421084
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}. The current
+ * implementation only handles conflicts within a single job. Users should avoid scheduling
+ * maintenance for the same table in different Flink jobs.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ if (nextEvaluationTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextEvaluationTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The recovery lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The recovery is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // A task is already running, waiting for it to finish
+ LOG.info("Delaying task on failed lock check: {}", current);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699704601
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
Review Comment:
`startsFrom == 0` means that we are finished this check loop
`startsFrom == evaluators.size()` means that we need to continue this check loop from the beginning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1700306350
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(Table table, String lockKey) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockKey, "Lock key should not be null");
+ this.table = table;
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
Review Comment:
> > also how do we deal with orphaned locks where unlock failed (e.g. due to network issue with catalog service)
>
> We could reuse the recovery lock mechanism for this situation: After a given period, if we can't acquire the lock, then we can create a recovery lock, and send a `Trigger.cleanup`. If the recovery lock is not removed after a time period we could just throw an exception.
I would leave this for another PR, as this is a non-trivial feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705729180
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTagBasedLockFactory.java:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TagBasedLockFactory.RUNNING_TAG_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.CommitFailedException;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTagBasedLockFactory extends TestLockFactoryBase {
Review Comment:
I think if we remove the TagBasedLockFactory, we will not need the SQL stuff, so we can revisit this after the decision there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1637636105
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
> tag-based locks also have higher overhead (metadata JSON file rewrite) compared to other standard solution like SQL database
Agreed. That's why the interface. We can provide other implementations later. Maybe even a REST catalog based solution, or HMS based solution above the JDBC based one. I envision this as a first solution without additional external depenency which could be used with any existing Iceberg implementation.
> also how are we going to deal with orphaned locks, where unlock didn't happen?
Job start with the cleanup will always delete the lock, and the job restart logic will fix any remaining locks.
Maybe we can add a lock timeout function too, so if we wait more than 1 day (configurable) for the lock then we just assume that it's orphaned, and then we reuse the same cleanup logic that used during the restart to clear the orphaned lock. I think this decision could be postponed until we see if this is a frequent issue, and simple monitoring and restart is not enough to solve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698436916
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698470388
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
Review Comment:
Added close to the LockManagerFactory
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
Review Comment:
Closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1726594832
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockFactoryBase.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+abstract class TestLockFactoryBase {
+ protected TriggerLockFactory lockFactory;
+
+ abstract TriggerLockFactory lockFactory();
+
+ @BeforeEach
+ void before() {
+ this.lockFactory = lockFactory();
+ lockFactory.open();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testTryLock() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock1.tryLock()).isFalse();
+ assertThat(lock2.tryLock()).isFalse();
+ }
+
+ @Test
+ void testUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ @Test
+ void testNoConflict() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createRecoveryLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock2.tryLock()).isTrue();
+ }
+
+ @Test
+ void testDoubleUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ lock.unlock();
Review Comment:
I’m a reluctant to add a return value to the `unlock` just for testing purposes.
If we want to ensure that the lock is removed, we can add a `tryLock` call after the `unlock`, but I think that is covered by other scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1700556989
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.UUID;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ @VisibleForTesting static final String RUNNING_TAG_PREFIX = "__flink_maintenance_running_";
+ @VisibleForTesting static final String RECOVERING_TAG_PREFIX = "__flink_maintenance_recovering_";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG_PREFIX);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG_PREFIX);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockPrefix;
+
+ public Lock(Table table, String lockPrefix) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockPrefix, "Lock key should not be null");
+ this.table = table;
+ this.lockPrefix = lockPrefix;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held. The relevant tag: {}", findLock());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ String lockKey = lockPrefix + UUID.randomUUID();
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ unused -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.createTag(lockKey, table.currentSnapshot().snapshotId());
+ manage.commit();
+ LOG.debug("Lock created");
+ } catch (Exception e) {
+ if (!isHeld()) {
+ LOG.warn("Retrying lock after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock created, hiding exception", e);
+ }
+ }
+ });
+ } catch (Exception e) {
+ LOG.info("Concurrent lock created. Is there a concurrent maintenance job running?", e);
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ public void unlock() {
+ table.refresh();
+
+ Optional<String> lockKey = findLock();
+ if (lockKey.isPresent()) {
+ Tasks.foreach(lockKey.get())
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ key -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.removeTag(key);
+ manage.commit();
+ } catch (Exception e) {
+ table.refresh();
+ if (table.refs().containsKey(key)) {
+ LOG.warn("Retrying lock removal after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock removed, hiding exception", e);
+ }
+ }
+ });
+ LOG.debug("Lock removed");
+ } else {
+ LOG.warn(
+ "Missing lock, can not remove. Found only the following tags: {}.",
+ table.refs().keySet());
+ }
+ }
+
+ @Override
+ public boolean isHeld() {
+ table.refresh();
+ return findLock().isPresent();
+ }
+
+ private Optional<String> findLock() {
+ return table.refs().keySet().stream().filter(tag -> tag.startsWith(lockPrefix)).findAny();
Review Comment:
we should probably throw an exception if there are multiple locks/tags exist for the prefix. because this is the assumption of how the random suffix works, let's enforce it here.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}. The current
Review Comment:
nit: import the class
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
Review Comment:
nit: maybe `// Simulate the action of removing lock and recoveryLock by downstream lock cleaner when it received recovery trigger`
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
Review Comment:
do we need to another table change event here?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
Review Comment:
> Added the check to the init shouldRestoreTasks as well
I am not seeing the change. not sure if I missed sth or not.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.UUID;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ @VisibleForTesting static final String RUNNING_TAG_PREFIX = "__flink_maintenance_running_";
+ @VisibleForTesting static final String RECOVERING_TAG_PREFIX = "__flink_maintenance_recovering_";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG_PREFIX);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG_PREFIX);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockPrefix;
+
+ public Lock(Table table, String lockPrefix) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockPrefix, "Lock key should not be null");
+ this.table = table;
+ this.lockPrefix = lockPrefix;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held. The relevant tag: {}", findLock());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ String lockKey = lockPrefix + UUID.randomUUID();
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ unused -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.createTag(lockKey, table.currentSnapshot().snapshotId());
+ manage.commit();
+ LOG.debug("Lock created");
+ } catch (Exception e) {
+ if (!isHeld()) {
+ LOG.warn("Retrying lock after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock created, hiding exception", e);
+ }
+ }
+ });
+ } catch (Exception e) {
+ LOG.info("Concurrent lock created. Is there a concurrent maintenance job running?", e);
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ public void unlock() {
+ table.refresh();
+
+ Optional<String> lockKey = findLock();
Review Comment:
let's add comment to explain that there could never been two locks created at the same time. `tryLock` is only called by single-parallelism `TriggerManager`, which always check if there is any existing lock or not before creating a new tag/lock.
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTagBasedLockFactory.java:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TagBasedLockFactory.RUNNING_TAG_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.CommitFailedException;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTagBasedLockFactory extends TestLockFactoryBase {
Review Comment:
`TestLockFactoryBase` extends from `OperatorTestBase`. is operator needed to test `TagBasedLockFactory`?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}. The current
+ * implementation only handles conflicts within a single job. Users should avoid scheduling
+ * maintenance for the same table in different Flink jobs.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ if (nextEvaluationTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextEvaluationTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The recovery lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The recovery is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // A task is already running, waiting for it to finish
+ LOG.info("Delaying task on failed lock check: {}", current);
Review Comment:
nit: maybe `"Failed to acquire lock. Delaying task to {}", current + lockCheckDelayMs`
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
Review Comment:
do we need this? Does timeout/expire depends on the processElement action?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/MetricsReporterFactoryForTests.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.MetricReporterFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+
+public class MetricsReporterFactoryForTests implements MetricReporterFactory {
Review Comment:
can we use `CollectingMetricsReporter` from Flink?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
Review Comment:
can you add a comment to explain why we have 2 triggers in this case?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
Review Comment:
can you add a comment to explain this if section?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
Review Comment:
typo? log -> lock
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}. The current
+ * implementation only handles conflicts within a single job. Users should avoid scheduling
+ * maintenance for the same table in different Flink jobs.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
Review Comment:
maybe add some info log for both if and else case.
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
Review Comment:
nit: is `locked` more accurate than `tableLock`?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
Review Comment:
why would both tasks triggered in this case?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService) throws Exception {
+ if (!inited) {
+ long current = timerService.currentProcessingTime();
+
+ // Initialize from state
+ this.nextEvaluationTime = nextEvaluationTimeState.value();
Review Comment:
`initializeState` happens before `open` method. we probably should move all state related to the `initializeState`
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
Review Comment:
> `startsFrom == 0` means that we are finished this check loop `startsFrom == evaluators.size()` means that we need to continue this check loop from the beginning
where do we use that distinguish?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
Review Comment:
maybe add one more step of advancing the time so that next check will still make the task as blocked?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
Review Comment:
this function name is difficult to see what is asserted. a couple of possible alternatives
1) make it more specific like `assertRateLimitedMetric(int expected)`
2) still one metric a time, but a more general form `assertMetric(String name, int expected)`
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
Review Comment:
a semantic question. in this case, would we call it `concurrent run`? also would be be `blocked` or `triggered`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701481389
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
Review Comment:
We have 2 tasks scheduled in this test:
- The firsts task is triggered on ever 2nd commit - this should be triggered on the 4th commit
- The second task is triggered on every 4th commit - this should be triggered on the 4th commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639299474
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
If you only have 2 operators with parallelism 1 and they share the slot sharing group, they will be scheduled in the same slot. And that means also that they will be on the same JVM.
I don't know how much this is guaranteed in the long run. Depends on how we define the concept of slot sharing group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639261053
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 3), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 10), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 0);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 1, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(5, 7, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 1, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 3, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 5, 7, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 1, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(3, 1, 0, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 2, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 10, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = new TableChange(1, 0, 0, 0, 1);
+
+ // Wait for one trigger
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for a second trigger
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(new TableChange(1, 0, 0, 0, 1), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 1), EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues())
+ .isEqualTo(Lists.newArrayList(Trigger.cleanUp(testHarness.getProcessingTime())));
+
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testCleanup(boolean cleanup) throws Exception {
+ // Simulate a lock remaining from a previous run
+ assertThat(lock.tryLock()).isTrue();
+
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, 1, cleanup);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ assertThat(lock.isHeld()).isEqualTo(!cleanup);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.cleanUp(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L,
+ false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private static Stream<Arguments> parametersForTestRecovery() {
+ return Stream.of(
+ Arguments.of(true, false),
+ Arguments.of(true, false),
+ Arguments.of(false, true),
+ Arguments.of(false, false));
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED,
+ rateLimiterTrigger)
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED,
+ concurrentRunTrigger)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ lock.unlock();
Review Comment:
The `TriggerManager` creates the lock before emitting the `Trigger`.
In the real use-case the lock will be removed by the last operator of the Table Maintenance Stream, to allow the `TriggerManager` to fire the next `Trigger`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1643989763
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
Review Comment:
Added the builder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1637686857
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/Trigger.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+@Internal
+class Trigger {
+ private final long timestamp;
+ private final SerializableTable table;
+ private final Integer taskId;
+ private final boolean isCleanUp;
+
+ private Trigger(long timestamp, SerializableTable table, Integer taskId, boolean isCleanUp) {
+ this.timestamp = timestamp;
+ this.table = table;
+ this.taskId = taskId;
Review Comment:
> in the 2nd option, why does watermark generator needs to be n instances?
Here is how lock handling will go:
- `TriggerManager` creates a lock
- We generate a Watermark which travels through all of the Maintenance Task streams
- We join the streams at the end and remove the lock when the Watermark arrives from all of the substreams.
We need to join the Maintenance Task streams, so when cleaning the locks on recovery then the recovery trigger watermark will signal that the previously started Maintenance Tasks are finished (whichever one was running).
In the 2nd option we have a sideoutput for every Maintenance Task. We need to put the same watermark to them, so the lock cleaner will receive a watermark from each of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724698190
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
+ return conn.prepareStatement(CREATE_LOCK_TABLE_SQL).execute();
+ });
+
+ } catch (SQLTimeoutException e) {
+ throw new UncheckedSQLException(
+ e, "Cannot initialize JDBC table maintenance lock: Query timed out");
+ } catch (SQLTransientConnectionException | SQLNonTransientConnectionException e) {
+ throw new UncheckedSQLException(
+ e, "Cannot initialize JDBC table maintenance lock: Connection failed");
+ } catch (SQLException e) {
+ throw new UncheckedSQLException(e, "Cannot initialize JDBC table maintenance lock");
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted in call to initialize");
+ }
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final JdbcClientPool pool;
+ private final String issuer;
+ private final Type type;
+
+ public Lock(JdbcClientPool pool, String issuer, Type type) {
+ this.pool = pool;
+ this.issuer = issuer;
+ this.type = type;
+ }
+
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ try {
+ return pool.run(
+ conn -> {
+ try (PreparedStatement sql = conn.prepareStatement(CREATE_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ sql.setString(3, UUID.randomUUID().toString());
+ return sql.executeUpdate() == 1;
+ }
+ });
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted during tryLock");
+ } catch (SQLException e) {
+ // SQL exception happened when creating the lock
+ throw new UncheckedSQLException(e, "Failed to create %s lock", type);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:NestedTryDepth")
+ @Override
+ public boolean isHeld() {
+ try {
+ return pool.run(
+ conn -> {
+ try (PreparedStatement sql = conn.prepareStatement(GET_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ try (ResultSet rs = sql.executeQuery()) {
+ return rs.next();
+ }
+ }
+ });
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted during isHeld");
+ } catch (SQLException e) {
+ // SQL exception happened when getting lock information
+ throw new UncheckedSQLException(e, "Failed to get lock information for %s", type);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:NestedTryDepth")
+ @Override
+ public void unlock() {
+ try {
+ pool.run(
+ conn -> {
+ String lockId;
+ try (PreparedStatement sql = conn.prepareStatement(GET_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ try (ResultSet rs = sql.executeQuery()) {
+ if (rs.next()) {
+ lockId = rs.getString(1);
Review Comment:
The `tryLock` and the `unlock` will happen on a different object instance, possibly on a different JVM even. So that doesn't help. We need to fetch the id in the `unlock` method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1726213077
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockFactoryBase.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+abstract class TestLockFactoryBase {
+ protected TriggerLockFactory lockFactory;
+
+ abstract TriggerLockFactory lockFactory();
+
+ @BeforeEach
+ void before() {
+ this.lockFactory = lockFactory();
+ lockFactory.open();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testTryLock() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock1.tryLock()).isFalse();
+ assertThat(lock2.tryLock()).isFalse();
+ }
+
+ @Test
+ void testUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ @Test
+ void testNoConflict() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createRecoveryLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock2.tryLock()).isTrue();
+ }
+
+ @Test
+ void testDoubleUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ lock.unlock();
Review Comment:
unlock can return false when the lock doesn't exist. then here we can assert that 1st unlock should return true and 2nd unlock should return false.
similar to the `LockManager` interface from api module.
```
/**
* Release a lock
*
* <p>exception must not be thrown for this method.
*
* @param entityId ID of the entity to lock
* @param ownerId ID of the owner if the lock
* @return if the owner held the lock and successfully released it.
*/
boolean release(String entityId, String ownerId);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724712629
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
+ return conn.prepareStatement(CREATE_LOCK_TABLE_SQL).execute();
Review Comment:
I don't see when this could be a failure and there would be no exception thrown. We could change the return null instead of the output of the execute, as we need to return something from the `run` method anyway, but I don't see a point adding more code here than neccessary.
WDYT?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
+ return conn.prepareStatement(CREATE_LOCK_TABLE_SQL).execute();
Review Comment:
I don't see when this could be a failure and there would be no exception thrown. We could change the return null instead of the output of the execute, as we need to return something from the `run` method anyway, but I don't see a point adding more code here than necessary.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724745652
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
Review Comment:
I think that would be misleading. The entityId is more-or-less descriptive, but the ownerId is different, as this lock doesn't really care about the ownership at all.
I opted for renaming them differently, which might be better:
- LOCK_TYPE
- LOCK_ID - previously `ISSUER`
- INSTANCE_ID - previously `LOCK_ID`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698429306
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JVMBasedLockFactory.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.concurrent.Semaphore;
+import org.apache.flink.annotation.Internal;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The locks are based on static {@link Semaphore} objects. We expect that the {@link
+ * TriggerManager} and the LockRemover operators will be placed on the same TaskManager (JVM), as
+ * they are both global. In this case JVM based locking should be enough to allow communication
+ * between the operators.
+ */
+@Internal
+public class JVMBasedLockFactory implements TriggerLockFactory {
Review Comment:
Removed the JVM lock altogether
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698467590
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ table.refresh();
+ Map<String, SnapshotRef> refs = table.refs();
Review Comment:
removed refresh, and also refs.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ table.refresh();
+ Map<String, SnapshotRef> refs = table.refs();
+ if (isHeld()) {
+ LOG.info("Lock is already held by someone: {}", refs.keySet());
Review Comment:
Removed the suggested parts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698483483
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
Review Comment:
As discussed on the other comment, we need this, so the scheduling is fair
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#issuecomment-2255494823
Hi @stevenzwu,
I would like to move forward with this PR.
I see the following open questions:
1. Lock implementation - I still think it is important to provide a working implementation without adding a requirement for external systems. For this the `Tag` based solution is working for every Iceberg deployment and there were no objections or better suggestions for it during the discussion on the mailing list (https://lists.apache.org/thread/vjf8m5wg840o58yz4y3q35k2mfhbm49l). Also tried to ask help from the Flink mailing list for alternative solutions, but I got a single one offline which turned out a dead end (https://lists.apache.org/thread/3ss83w1hp8wpocx13kcqr7qyjnghvnc8). - **I propose to remove the JVM based lock, and move forward with the Tag based lock**
2. TableChange parameters - I am open to discussing the list of parameters which will drive the triggers. It will touch a big part of the previous PR as well, and will make this change even harder to review/read. - **I propose to do it in the next PR**
3. There were some questions about the overall architecture. This is described in the original proposal doc [here](https://docs.google.com/document/d/16g3vR18mVBy8jbFaLjf2JwAANuYOmIwr15yDDxovdnA/edit#heading=h.5dc7u19et6mc). - **I'm open to discuss the proposal offline too, if that is needed.**
WDYT @stevenzwu? Could we move forward along these lines, or do you have another proposal? Do you have any foundational concerns about the PR?
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1726304565
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
+ return conn.prepareStatement(CREATE_LOCK_TABLE_SQL).execute();
Review Comment:
my mistake. I thought the boolean return value from `execute` is success or failure. it is not.
```
true if the first result is a ResultSet object; false if the first result is an update count or there is no result
```
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
Review Comment:
agree that `OWNER_ID` is misleading now I can remember the purpose of this field after seeing the code comment on the race condition, which helped me refresh my memory.
current naming LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1726315077
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+
+ @VisibleForTesting
+ static final String INIT_LOCK_TABLES_PROPERTY = "flink-maintenance.lock.jdbc.init-lock-tables";
+
+ private static final String LOCK_TABLE_NAME = "flink_maintenance_lock";
+ private static final int LOCK_ID_MAX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ String.format(
+ "CREATE TABLE %s "
+ + "(LOCK_TYPE CHAR(1) NOT NULL, "
+ + "LOCK_ID VARCHAR(%s) NOT NULL, "
+ + "INSTANCE_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, LOCK_ID))",
+ LOCK_TABLE_NAME, LOCK_ID_MAX_LENGTH);
+
+ private static final String CREATE_LOCK_SQL =
+ String.format(
+ "INSERT INTO %s (LOCK_TYPE, LOCK_ID, INSTANCE_ID) VALUES (?, ?, ?)", LOCK_TABLE_NAME);
+ private static final String GET_LOCK_SQL =
+ String.format("SELECT INSTANCE_ID FROM %s WHERE LOCK_TYPE=? AND LOCK_ID=?", LOCK_TABLE_NAME);
+ private static final String DELETE_LOCK_SQL =
+ String.format(
+ "DELETE FROM %s WHERE LOCK_TYPE=? AND LOCK_ID=? AND INSTANCE_ID=?", LOCK_TABLE_NAME);
+
+ private final String uri;
+ private final String lockId;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ /**
+ * Creates a new {@link TriggerLockFactory}. The issuer should be unique between the users of the
Review Comment:
nit: `issuer` -> `lockId`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1637170443
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
Review Comment:
still not quite following. I thought TriggerManager accumulates `TableChange` and decide if it is time to actually trigger/schedule the maintenance task. should the accumulation be done via `List<TableChange>` or on-the-fly merged single `TableChange`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1637109731
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
Review Comment:
I saw `ProcessFunction#Context` also has `timerService()` available. `getRuntimeContext()` also seems to work for `ProcessFunction`
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
also how are we going to deal with orphaned locks, where `unlock` didn't happen?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
Looking at the `PropertiesUpdate` code, it only provides `set` (not `putIfAbsent`). so maybe it doesn't provide the atomicity.
My early comment regarding using tag for lock still holds true though. it doesn't feel very natural. I understand the benefit of using Iceberg construct for no additional dependency. But tag-based locks also have higher overhead (metadata JSON file rewrite) compared to other standard solution like SQL database. Iceberg commit's atomicity guarantee in the end also comes from SQL database.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ table.refresh();
+ Map<String, SnapshotRef> refs = table.refs();
+ if (isHeld()) {
+ LOG.info("Lock is already held by someone: {}", refs.keySet());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ unused -> {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.createTag(lockKey, table.currentSnapshot().snapshotId());
+ manage.commit();
+ LOG.debug("Lock created");
+ });
+ } catch (Exception e) {
Review Comment:
what if it is a error (like OutOfMemoryError)?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
Review Comment:
oh. you move the init logic here (instead of `initializeState`) due to the need of `currentProcessingTime()`?
nit: maybe a bit cleaned to move the method to the first line in this method, e.g. `current` can be calculated inside the init method.
```
init(out, ctx.timerService())
```
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/Trigger.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+@Internal
+class Trigger {
+ private final long timestamp;
+ private final SerializableTable table;
+ private final Integer taskId;
+ private final boolean isCleanUp;
+
+ private Trigger(long timestamp, SerializableTable table, Integer taskId, boolean isCleanUp) {
+ this.timestamp = timestamp;
+ this.table = table;
+ this.taskId = taskId;
Review Comment:
I don't fully understand the 2 options. we can postpone this discussion to later PRs when the taskId is actually used.
in the 2nd option, why does watermark generator needs to be n instances?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
Review Comment:
still not quite following. I thought TriggerManager accumulates `TableChange` and decide if it is time to actually trigger/schedule the maintenance task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1643992382
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
Review Comment:
Removed the `clearLocks` for now. We might want to get it back when/if we start supporting concurrent compaction jobs, but it is not important as of now. The recovery will create it's own lock anyway, and cleanup will remove the lock in the end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1643989141
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
Review Comment:
Moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1643988437
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerEvaluator.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.List;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Internal
+class TriggerEvaluator implements Serializable {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerEvaluator.class);
+ private final List<Predicate> predicates;
+
+ private TriggerEvaluator(List<Predicate> predicates) {
+ Preconditions.checkArgument(!predicates.isEmpty(), "Provide at least 1 condition.");
+
+ this.predicates = predicates;
+ }
+
+ boolean check(TableChange event, long lastTime, long currentTime) {
Review Comment:
Added `Ms` to the names
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639268169
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
Review Comment:
All of the publicly available parameters of these operators will be set by the `TableMaintenance.Builder`.
That said, this config is probably not needed anymore. The original intent was to enable the user to remove stale locks on job restart, but the new recovery solution covers this possibility.
Let me think a bit more on this, but I might end up removing this configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698472738
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
Review Comment:
Renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701468373
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
Review Comment:
Renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701438248
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/MetricsReporterFactoryForTests.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.MetricReporterFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+
+public class MetricsReporterFactoryForTests implements MetricReporterFactory {
Review Comment:
Any idea, on how to get the current `MetricsReporter` in the tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1707557043
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerLockFactory.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.Closeable;
+import java.io.Serializable;
+import org.apache.flink.annotation.Experimental;
+
+/** Lock interface for handling locks for the Flink Table Maintenance jobs. */
+@Experimental
+public interface TriggerLockFactory extends Serializable, Closeable {
+ void open();
+
+ Lock createLock();
+
+ Lock createRecoveryLock();
+
+ interface Lock {
+ /**
+ * Tries to acquire a lock with a given key. Anyone already holding a lock would prevent
+ * acquiring this lock. Not reentrant.
+ *
+ * <p>Called by {@link TriggerManager}. Implementations could assume that are no concurrent
+ * calls for this method.
+ *
+ * @return <code>true</code> if the lock is acquired by this job, <code>false</code> if the lock
+ * is already held by someone
+ */
+ boolean tryLock();
+
+ /**
+ * Checks if the lock is already taken.
+ *
+ * @return <code>true</code> if the lock is held by someone
+ */
+ boolean isHeld();
+
+ // TODO: Fix the link to the LockRemover when we have a final name and implementation
+ /**
+ * Releases the lock. Should not fail if the lock is not held by anyone.
+ *
+ * <p>Called by LockRemover. Implementations could assume that are no concurrent calls for this
+ * method.
+ */
+ void unlock();
+ }
Review Comment:
Why are creating new interfaces rather than enhancing the existing ones ?
https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/LockManager.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1697278353
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
Review Comment:
Yes. We make sure that only on timer is scheduled to prevent race conditions and such.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1697274311
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ table.refresh();
+ Map<String, SnapshotRef> refs = table.refs();
+ if (isHeld()) {
+ LOG.info("Lock is already held by someone: {}", refs.keySet());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
Review Comment:
Also this happens only very-very rarely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1695810956
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
Review Comment:
should we load the table once in the factory constructor? lock object only needs to refresh table state in the tryLock and unlock methods.
also it doesn't seem that `tableLoader` is closed.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ table.refresh();
+ Map<String, SnapshotRef> refs = table.refs();
+ if (isHeld()) {
+ LOG.info("Lock is already held by someone: {}", refs.keySet());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
Review Comment:
if table is empty, is there any need to run maintenance. wondering if retiring false makes more sense here. is the purpose to avoid wait loop on `tryLock` to be true.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ table.refresh();
+ Map<String, SnapshotRef> refs = table.refs();
+ if (isHeld()) {
+ LOG.info("Lock is already held by someone: {}", refs.keySet());
Review Comment:
nit: remove `someone`. I don't know if we need to print the refs keySet. it would imply all tags are locks
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ table.refresh();
+ Map<String, SnapshotRef> refs = table.refs();
Review Comment:
`refs` is retrieved both here and in the `isHeld` method. `table.refresh()` is also done in the `isHeld` method. should we remove line 76 and 77?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -83,7 +96,7 @@ class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
Preconditions.checkArgument(
- lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
Review Comment:
nit: `1 ms` with a space
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
Review Comment:
do we need the local variable of `nextTime`? read `nextEvaluationTime` directly?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
if it is hard to guarantee colocation, I would suggest we remove the JVM lock for now. Otherwise users may misuse it without knowing the problem. basically not expose the JVM lock until we can guarantee colocation.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
Review Comment:
we could have multiple timers scheduled for this scenario if the recoveryLock is not released in multiple `checkAndFire` calls?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
Review Comment:
> When there is nothing to trigger, we start from the beginning, as the order of the tasks might be important (RewriteDataFiles first, and then RewriteManifestFiles later - in this case we don't want to rewrite manifest files which are removed by the RewriteDataFiles anyways)
might make sense to add as code comment?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
Review Comment:
where do we need to differentiate the 2 situations?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 3), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 10), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 0);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 1, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(5, 7, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 1, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 3, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 5, 7, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 1, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(3, 1, 0, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 2, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 10, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = new TableChange(1, 0, 0, 0, 1);
+
+ // Wait for one trigger
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for a second trigger
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(new TableChange(1, 0, 0, 0, 1), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 1), EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues())
+ .isEqualTo(Lists.newArrayList(Trigger.cleanUp(testHarness.getProcessingTime())));
+
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testCleanup(boolean cleanup) throws Exception {
+ // Simulate a lock remaining from a previous run
+ assertThat(lock.tryLock()).isTrue();
+
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, 1, cleanup);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ assertThat(lock.isHeld()).isEqualTo(!cleanup);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.cleanUp(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L,
+ false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private static Stream<Arguments> parametersForTestRecovery() {
+ return Stream.of(
+ Arguments.of(true, false),
+ Arguments.of(true, false),
+ Arguments.of(false, true),
+ Arguments.of(false, false));
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED,
+ rateLimiterTrigger)
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED,
+ concurrentRunTrigger)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ lock.unlock();
Review Comment:
can we add those as code comment?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
Review Comment:
where is tableLoader closed?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
Review Comment:
maybe add a comment like `Recovered tasks in progress. Skip trigger check`?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
Review Comment:
reset `isCleanUp` to false when recovery is done? `isCleanUp` is only used once by the `init` method. why don't just use the `inited` boolean?
maybe `isCleanUp` naming doesn't help either. seems more like `shouldRestoreTasks`?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/Trigger.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+@Internal
+class Trigger {
+ private final long timestamp;
+ private final SerializableTable table;
+ private final Integer taskId;
+ private final boolean isCleanUp;
+
+ private Trigger(long timestamp, SerializableTable table, Integer taskId, boolean isCleanUp) {
+ this.timestamp = timestamp;
+ this.table = table;
+ this.taskId = taskId;
Review Comment:
ok. let' revisit this once we see later parts
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
Review Comment:
is the assumption here that there is only one timer scheduled at the most?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
Review Comment:
nit: Iceberg error msg tends to be in the style of `Invalid something: reason`. so here could be `Invalid evaluators: null or empty`
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
Review Comment:
didn't quite understand the race condition part
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService(), timestamp);
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime.value());
+ }
+
+ private void schedule(TimerService timerService, long time) throws IOException {
+ nextEvaluationTime.update(time);
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService, long current)
+ throws Exception {
+ if (!inited) {
+ // Initialize with empty changes and current timestamp
+ if (!accumulatedChanges.get().iterator().hasNext()) {
+ List<TableChange> changes = Lists.newArrayListWithCapacity(evaluators.size());
+ List<Long> triggerTimes = Lists.newArrayListWithCapacity(evaluators.size());
+ for (int i = 0; i < evaluators.size(); ++i) {
+ changes.add(TableChange.empty());
+ triggerTimes.add(current);
+ }
+
+ accumulatedChanges.update(changes);
+ lastTriggerTimes.update(triggerTimes);
+ }
+
+ if (isCleanUp) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a cleanup lock
+ // - fire a clean-up trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end.
+ recoveryLock.tryLock();
Review Comment:
add the context as code comment as it is not immediately obvious?
if tryLock failed, it is fine to send the `Trigger.cleanUp` event?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698465204
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(tableLoader, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(tableLoader, RECOVERING_TAG);
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(TableLoader tableLoader, String lockKey) {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
Review Comment:
Added open/close to the LockFactory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699709058
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
Review Comment:
Yeah, fixed the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699663282
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(Table table, String lockKey) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockKey, "Lock key should not be null");
+ this.table = table;
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
Review Comment:
> also how do we deal with orphaned locks where unlock failed (e.g. due to network issue with catalog service)
We could reuse the recovery lock mechanism for this situation: After a given period, if we can't acquire the lock, then we can create a recovery lock, and send a `Trigger.cleanup`. If the recovery lock is not removed after a time period we could just throw an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699587030
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(Table table, String lockKey) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockKey, "Lock key should not be null");
+ this.table = table;
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ unused -> {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.createTag(lockKey, table.currentSnapshot().snapshotId());
+ manage.commit();
+ LOG.debug("Lock created");
+ });
+ } catch (Exception e) {
+ LOG.info("Concurrent lock created. Stop concurrent maintenance jobs.", e);
Review Comment:
Would this message be more user friendly:
```
Concurrent lock created. Is there a concurrent maintenance job running?
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701410482
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService) throws Exception {
+ if (!inited) {
+ long current = timerService.currentProcessingTime();
+
+ // Initialize from state
+ this.nextEvaluationTime = nextEvaluationTimeState.value();
Review Comment:
We can't access the keyed state in the `initializeState` method:
```
java.lang.NullPointerException: Keyed state 'triggerManagerNextTriggerTime' with type VALUE can only be used on a 'keyed stream', i.e., after a 'keyBy()' operation.
at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:76)
at org.apache.flink.streaming.api.operators.StreamingRuntimeContext.checkPreconditionsAndGetKeyedStateStore(StreamingRuntimeContext.java:238)
at org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getState(StreamingRuntimeContext.java:201)
at org.apache.iceberg.flink.maintenance.operator.TriggerManager.initializeState(TriggerManager.java:158)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701493223
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
Review Comment:
This is also testing that there are no other "random" counters there.
Also checking that the `TRIGGERED` counter is 1, and the `NOTHING_TO_TRIGGER` counter is 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1700302146
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService) throws Exception {
+ if (!inited) {
+ long current = timerService.currentProcessingTime();
+
+ // Initialize from state
+ this.nextEvaluationTime = nextEvaluationTimeState.value();
Review Comment:
We initialize the state-s in the `open` method. The `initializeState` is called after that.
That is why I needed to move the state restoration to here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639284858
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService(), timestamp);
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime.value());
+ }
+
+ private void schedule(TimerService timerService, long time) throws IOException {
+ nextEvaluationTime.update(time);
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService, long current)
+ throws Exception {
+ if (!inited) {
+ // Initialize with empty changes and current timestamp
+ if (!accumulatedChanges.get().iterator().hasNext()) {
+ List<TableChange> changes = Lists.newArrayListWithCapacity(evaluators.size());
+ List<Long> triggerTimes = Lists.newArrayListWithCapacity(evaluators.size());
+ for (int i = 0; i < evaluators.size(); ++i) {
+ changes.add(TableChange.empty());
+ triggerTimes.add(current);
+ }
+
+ accumulatedChanges.update(changes);
+ lastTriggerTimes.update(triggerTimes);
+ }
+
+ if (isCleanUp) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a cleanup lock
+ // - fire a clean-up trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end.
+ recoveryLock.tryLock();
Review Comment:
In the last operator of the Table Maintenance Stream, where we join the results of all of the Maintenance Tasks, and remove the lock when the Watermark has arrived from all of them. This makes sure that any recovered and still running task is finished at this time.
We essentially use the Watermark as a marker that every record is processed in all of the Maintenance Tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1643990192
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 3), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 10), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 0);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 1, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(5, 7, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 1, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 3, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 5, 7, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 1, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(3, 1, 0, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 2, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 10, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = new TableChange(1, 0, 0, 0, 1);
+
+ // Wait for one trigger
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for a second trigger
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(new TableChange(1, 0, 0, 0, 1), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 1), EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues())
+ .isEqualTo(Lists.newArrayList(Trigger.cleanUp(testHarness.getProcessingTime())));
+
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testCleanup(boolean cleanup) throws Exception {
+ // Simulate a lock remaining from a previous run
+ assertThat(lock.tryLock()).isTrue();
+
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, 1, cleanup);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ assertThat(lock.isHeld()).isEqualTo(!cleanup);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.cleanUp(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L,
+ false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private static Stream<Arguments> parametersForTestRecovery() {
+ return Stream.of(
+ Arguments.of(true, false),
+ Arguments.of(true, false),
+ Arguments.of(false, true),
+ Arguments.of(false, false));
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED,
+ rateLimiterTrigger)
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED,
+ concurrentRunTrigger)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ lock.unlock();
+ }
+
+ private static TriggerManager manager(
+ TableLoader tableLoader,
+ TriggerEvaluator evaluator,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ return new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1),
+ Lists.newArrayList(evaluator),
+ minFireDelayMs,
+ lockCheckDelayMs,
+ clearLocks);
+ }
+
+ private static TriggerEvaluator triggerEvaluatorWith2Commits() {
Review Comment:
Refactored a bit these methods
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 3), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 10), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 0);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 1, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(5, 7, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 1, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 3, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 5, 7, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 1, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(3, 1, 0, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 2, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 10, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = new TableChange(1, 0, 0, 0, 1);
+
+ // Wait for one trigger
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for a second trigger
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(new TableChange(1, 0, 0, 0, 1), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 1), EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues())
+ .isEqualTo(Lists.newArrayList(Trigger.cleanUp(testHarness.getProcessingTime())));
+
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testCleanup(boolean cleanup) throws Exception {
+ // Simulate a lock remaining from a previous run
+ assertThat(lock.tryLock()).isTrue();
+
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, 1, cleanup);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ assertThat(lock.isHeld()).isEqualTo(!cleanup);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.cleanUp(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L,
+ false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private static Stream<Arguments> parametersForTestRecovery() {
+ return Stream.of(
+ Arguments.of(true, false),
+ Arguments.of(true, false),
+ Arguments.of(false, true),
+ Arguments.of(false, false));
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED,
+ rateLimiterTrigger)
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED,
+ concurrentRunTrigger)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ lock.unlock();
+ }
+
+ private static TriggerManager manager(
+ TableLoader tableLoader,
+ TriggerEvaluator evaluator,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ return new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1),
+ Lists.newArrayList(evaluator),
+ minFireDelayMs,
+ lockCheckDelayMs,
+ clearLocks);
+ }
+
+ private static TriggerEvaluator triggerEvaluatorWith2Commits() {
+ return new TriggerEvaluator.Builder().commitNumber(2).build();
+ }
+
+ private static TriggerManager manager(TableLoader tableLoader, TriggerEvaluator evaluator) {
+ return manager(tableLoader, evaluator, 1, 1, false);
+ }
+
+ private static TriggerManager manager(TableLoader tableLoader) {
Review Comment:
Refactored a bit these methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1722057725
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
Review Comment:
nit: `maintenance-` ==> `flink-maintenance`?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
Review Comment:
nit: `flink_maintenance_lock
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
Review Comment:
this should be `info` level
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
Review Comment:
nit: would be more readable to use `String.format`?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
+ return conn.prepareStatement(CREATE_LOCK_TABLE_SQL).execute();
+ });
+
+ } catch (SQLTimeoutException e) {
+ throw new UncheckedSQLException(
+ e, "Cannot initialize JDBC table maintenance lock: Query timed out");
+ } catch (SQLTransientConnectionException | SQLNonTransientConnectionException e) {
+ throw new UncheckedSQLException(
+ e, "Cannot initialize JDBC table maintenance lock: Connection failed");
+ } catch (SQLException e) {
+ throw new UncheckedSQLException(e, "Cannot initialize JDBC table maintenance lock");
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted in call to initialize");
+ }
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final JdbcClientPool pool;
+ private final String issuer;
+ private final Type type;
+
+ public Lock(JdbcClientPool pool, String issuer, Type type) {
+ this.pool = pool;
+ this.issuer = issuer;
+ this.type = type;
+ }
+
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ try {
+ return pool.run(
+ conn -> {
+ try (PreparedStatement sql = conn.prepareStatement(CREATE_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ sql.setString(3, UUID.randomUUID().toString());
+ return sql.executeUpdate() == 1;
+ }
+ });
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted during tryLock");
+ } catch (SQLException e) {
+ // SQL exception happened when creating the lock
+ throw new UncheckedSQLException(e, "Failed to create %s lock", type);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:NestedTryDepth")
+ @Override
+ public boolean isHeld() {
+ try {
+ return pool.run(
+ conn -> {
+ try (PreparedStatement sql = conn.prepareStatement(GET_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ try (ResultSet rs = sql.executeQuery()) {
+ return rs.next();
+ }
+ }
+ });
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted during isHeld");
+ } catch (SQLException e) {
+ // SQL exception happened when getting lock information
+ throw new UncheckedSQLException(e, "Failed to get lock information for %s", type);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:NestedTryDepth")
+ @Override
+ public void unlock() {
+ try {
+ pool.run(
+ conn -> {
+ String lockId;
+ try (PreparedStatement sql = conn.prepareStatement(GET_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ try (ResultSet rs = sql.executeQuery()) {
+ if (rs.next()) {
+ lockId = rs.getString(1);
Review Comment:
shouldn't the `lockId` be saved in the `lock()` step in order to avoid unlock a lock created by another process/thread?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
Review Comment:
add some Javadoc/comment to explain why `LOCK_ID` is needed and what kind of problem it is meant to solve?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockFactoryBase.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+abstract class TestLockFactoryBase {
+ protected TriggerLockFactory lockFactory;
+
+ abstract TriggerLockFactory lockFactory();
+
+ @BeforeEach
+ void before() {
+ this.lockFactory = lockFactory();
+ lockFactory.open();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testTryLock() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock1.tryLock()).isFalse();
+ assertThat(lock2.tryLock()).isFalse();
+ }
+
+ @Test
+ void testUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ @Test
+ void testNoConflict() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createRecoveryLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock2.tryLock()).isTrue();
+ }
+
+ @Test
+ void testDoubleUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ lock.unlock();
Review Comment:
I am thinking maybe `unlock` should also return a boolean value to indicate success or failure?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockFactoryBase.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+abstract class TestLockFactoryBase {
+ protected TriggerLockFactory lockFactory;
+
+ abstract TriggerLockFactory lockFactory();
+
+ @BeforeEach
+ void before() {
+ this.lockFactory = lockFactory();
+ lockFactory.open();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testTryLock() {
+ TriggerLockFactory.Lock lock1 = lockFactory.createLock();
+ TriggerLockFactory.Lock lock2 = lockFactory.createLock();
+ assertThat(lock1.tryLock()).isTrue();
+ assertThat(lock1.tryLock()).isFalse();
+ assertThat(lock2.tryLock()).isFalse();
+ }
+
+ @Test
+ void testUnLock() {
+ TriggerLockFactory.Lock lock = lockFactory.createLock();
+ assertThat(lock.tryLock()).isTrue();
+
+ lock.unlock();
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ @Test
+ void testNoConflict() {
Review Comment:
nit: this name is not very informative. maybe `testNoConflictWithRecoveryLock`?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
Review Comment:
nit: is `prefix` accurate here? just `ISSUER_MAX_LENGTH`?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
+ return conn.prepareStatement(CREATE_LOCK_TABLE_SQL).execute();
Review Comment:
check return value and log the execution result (success/failure)
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
Review Comment:
add some Javadoc for the constructor args. especially explain `issuer` in more details.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
Review Comment:
need to also check `properties` is not null, as it was passed directly to `JdbcClientPool`
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
Review Comment:
nit: debug level?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
+ return conn.prepareStatement(CREATE_LOCK_TABLE_SQL).execute();
+ });
+
+ } catch (SQLTimeoutException e) {
+ throw new UncheckedSQLException(
+ e, "Cannot initialize JDBC table maintenance lock: Query timed out");
+ } catch (SQLTransientConnectionException | SQLNonTransientConnectionException e) {
+ throw new UncheckedSQLException(
+ e, "Cannot initialize JDBC table maintenance lock: Connection failed");
+ } catch (SQLException e) {
+ throw new UncheckedSQLException(e, "Cannot initialize JDBC table maintenance lock");
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted in call to initialize");
+ }
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final JdbcClientPool pool;
+ private final String issuer;
+ private final Type type;
+
+ public Lock(JdbcClientPool pool, String issuer, Type type) {
+ this.pool = pool;
+ this.issuer = issuer;
+ this.type = type;
+ }
+
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ try {
+ return pool.run(
+ conn -> {
+ try (PreparedStatement sql = conn.prepareStatement(CREATE_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ sql.setString(3, UUID.randomUUID().toString());
+ return sql.executeUpdate() == 1;
+ }
+ });
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted during tryLock");
+ } catch (SQLException e) {
+ // SQL exception happened when creating the lock
+ throw new UncheckedSQLException(e, "Failed to create %s lock", type);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:NestedTryDepth")
+ @Override
+ public boolean isHeld() {
+ try {
+ return pool.run(
+ conn -> {
+ try (PreparedStatement sql = conn.prepareStatement(GET_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ try (ResultSet rs = sql.executeQuery()) {
+ return rs.next();
+ }
+ }
+ });
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted during isHeld");
+ } catch (SQLException e) {
+ // SQL exception happened when getting lock information
+ throw new UncheckedSQLException(e, "Failed to get lock information for %s", type);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:NestedTryDepth")
+ @Override
+ public void unlock() {
+ try {
+ pool.run(
+ conn -> {
+ String lockId;
+ try (PreparedStatement sql = conn.prepareStatement(GET_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ try (ResultSet rs = sql.executeQuery()) {
+ if (rs.next()) {
+ lockId = rs.getString(1);
+ } else {
+ return null;
+ }
+ }
+ }
+
+ try (PreparedStatement sql = conn.prepareStatement(DELETE_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ sql.setString(3, lockId);
+ sql.executeUpdate();
Review Comment:
check the return value and log the delete result at info level
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
Review Comment:
I looked at the `LockManager` class and think maybe we can learn from it. I like the naming of `entityId` (instead of `issuer`) and `ownerId` (instead of `lockId`).
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
Review Comment:
nit: `Creating Flink maintenance lock table {}`.
not sure about what `store Iceberg catalog tables` mean here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724703605
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
+ return conn.prepareStatement(CREATE_LOCK_TABLE_SQL).execute();
+ });
+
+ } catch (SQLTimeoutException e) {
+ throw new UncheckedSQLException(
+ e, "Cannot initialize JDBC table maintenance lock: Query timed out");
+ } catch (SQLTransientConnectionException | SQLNonTransientConnectionException e) {
+ throw new UncheckedSQLException(
+ e, "Cannot initialize JDBC table maintenance lock: Connection failed");
+ } catch (SQLException e) {
+ throw new UncheckedSQLException(e, "Cannot initialize JDBC table maintenance lock");
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted in call to initialize");
+ }
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final JdbcClientPool pool;
+ private final String issuer;
+ private final Type type;
+
+ public Lock(JdbcClientPool pool, String issuer, Type type) {
+ this.pool = pool;
+ this.issuer = issuer;
+ this.type = type;
+ }
+
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ try {
+ return pool.run(
+ conn -> {
+ try (PreparedStatement sql = conn.prepareStatement(CREATE_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ sql.setString(3, UUID.randomUUID().toString());
+ return sql.executeUpdate() == 1;
+ }
+ });
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted during tryLock");
+ } catch (SQLException e) {
+ // SQL exception happened when creating the lock
+ throw new UncheckedSQLException(e, "Failed to create %s lock", type);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:NestedTryDepth")
+ @Override
+ public boolean isHeld() {
+ try {
+ return pool.run(
+ conn -> {
+ try (PreparedStatement sql = conn.prepareStatement(GET_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ try (ResultSet rs = sql.executeQuery()) {
+ return rs.next();
+ }
+ }
+ });
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new UncheckedInterruptedException(e, "Interrupted during isHeld");
+ } catch (SQLException e) {
+ // SQL exception happened when getting lock information
+ throw new UncheckedSQLException(e, "Failed to get lock information for %s", type);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:NestedTryDepth")
+ @Override
+ public void unlock() {
+ try {
+ pool.run(
+ conn -> {
+ String lockId;
+ try (PreparedStatement sql = conn.prepareStatement(GET_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ try (ResultSet rs = sql.executeQuery()) {
+ if (rs.next()) {
+ lockId = rs.getString(1);
+ } else {
+ return null;
+ }
+ }
+ }
+
+ try (PreparedStatement sql = conn.prepareStatement(DELETE_LOCK_SQL)) {
+ sql.setString(1, type.key);
+ sql.setString(2, issuer);
+ sql.setString(3, lockId);
+ sql.executeUpdate();
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705645226
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
Review Comment:
Line 189 is calling `processElement`, and link 190 is checking that there is no output...
I might misunderstand your comment though... could you please elaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705732746
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1705727844
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService) throws Exception {
+ if (!inited) {
+ long current = timerService.currentProcessingTime();
+
+ // Initialize from state
+ this.nextEvaluationTime = nextEvaluationTimeState.value();
Review Comment:
We need the `KeyedProcessFunction` to access the `onTimer` method.
We need the `init` to initialize the `lastTriggerTimes` with the current time.
We also need the `init` to send the recovery trigger.
So the `init` method will be there and contain some part of the initialization. I would rather keep the whole initialization in a single place, than scatter that around multiple methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1713958997
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTagBasedLockFactory.java:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TagBasedLockFactory.RUNNING_TAG_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.CommitFailedException;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTagBasedLockFactory extends TestLockFactoryBase {
Review Comment:
Removed `TagBasedLockFactory`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1643987852
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
Created a `JVMBasedLockFactory` which uses `Semaphore`s as locks.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/MetricConstants.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+public class MetricConstants {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1697276904
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
Review Comment:
`nextEvaluationTime` might be changed by the `checkAndFire` method, and I prefer to print the `nextTime` in the logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701396639
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.UUID;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ @VisibleForTesting static final String RUNNING_TAG_PREFIX = "__flink_maintenance_running_";
+ @VisibleForTesting static final String RECOVERING_TAG_PREFIX = "__flink_maintenance_recovering_";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG_PREFIX);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG_PREFIX);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockPrefix;
+
+ public Lock(Table table, String lockPrefix) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockPrefix, "Lock key should not be null");
+ this.table = table;
+ this.lockPrefix = lockPrefix;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held. The relevant tag: {}", findLock());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ String lockKey = lockPrefix + UUID.randomUUID();
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ unused -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.createTag(lockKey, table.currentSnapshot().snapshotId());
+ manage.commit();
+ LOG.debug("Lock created");
+ } catch (Exception e) {
+ if (!isHeld()) {
+ LOG.warn("Retrying lock after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock created, hiding exception", e);
+ }
+ }
+ });
+ } catch (Exception e) {
+ LOG.info("Concurrent lock created. Is there a concurrent maintenance job running?", e);
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ public void unlock() {
+ table.refresh();
+
+ Optional<String> lockKey = findLock();
Review Comment:
Added a comment to the `Lock` methods, so other implementations could build on this assumption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701419852
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}. The current
+ * implementation only handles conflicts within a single job. Users should avoid scheduling
+ * maintenance for the same table in different Flink jobs.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701518242
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
Review Comment:
If the `startsFrom` is 0, then we did not execute anything in this loop.
If the `startsFrom` is anything but 0, it means we executed something. If we would execute the last task, and we calculate the `startsFrom` as `(startsFrom + 1) % size` then it would be 0.
```
// Nothing to execute
if (startsFrom == 0) {
nothingToTriggerCounter.inc();
LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698473585
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699712652
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService) throws Exception {
+ if (!inited) {
+ long current = timerService.currentProcessingTime();
+
+ // Initialize from state
+ this.nextEvaluationTime = nextEvaluationTimeState.value();
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ if (shouldRestoreTasks) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a cleanup lock
Review Comment:
changed the comment and also changed `Trigger.cleanUp` to `Trigger.recovery`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699687077
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724524981
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
+ private static final String LOCK_TABLE_NAME = "maintenance_lock";
+ private static final int MAX_PREFIX_LENGTH = 100;
+ private static final String CREATE_LOCK_TABLE_SQL =
+ "CREATE TABLE "
+ + LOCK_TABLE_NAME
+ + "(LOCK_TYPE CHAR(1) NOT NULL, ISSUER VARCHAR("
+ + MAX_PREFIX_LENGTH
+ + ") NOT NULL, LOCK_ID CHAR(36) NOT NULL, PRIMARY KEY (LOCK_TYPE, ISSUER))";
+ private static final String CREATE_LOCK_SQL =
+ "INSERT INTO " + LOCK_TABLE_NAME + " (LOCK_TYPE, ISSUER, LOCK_ID) VALUES (?, ?, ?)";
+ private static final String GET_LOCK_SQL =
+ "SELECT LOCK_ID FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=?";
+ private static final String DELETE_LOCK_SQL =
+ "DELETE FROM " + LOCK_TABLE_NAME + " WHERE LOCK_TYPE=? AND ISSUER=? AND LOCK_ID=?";
+
+ private final String uri;
+ private final String issuer;
+ private final Map<String, String> properties;
+ private transient JdbcClientPool pool;
+
+ public JdbcLockFactory(String uri, String issuer, Map<String, String> properties) {
+ Preconditions.checkNotNull(uri, "JDBC connection URI is required");
+ Preconditions.checkArgument(
+ issuer.length() < MAX_PREFIX_LENGTH,
+ "Invalid prefix length: issuer should be shorter than %s",
+ MAX_PREFIX_LENGTH);
+ this.uri = uri;
+ this.issuer = issuer;
+ this.properties = properties;
+ }
+
+ @Override
+ public void open() {
+ this.pool = new JdbcClientPool(1, uri, properties);
+
+ if (PropertyUtil.propertyAsBoolean(properties, INIT_LOCK_TABLES_PROPERTY, false)) {
+ initializeLockTables();
+ }
+ }
+
+ /** Only used in testing to share the jdbc pool */
+ @VisibleForTesting
+ void open(JdbcLockFactory other) {
+ this.pool = other.pool;
+ }
+
+ @Override
+ public Lock createLock() {
+ return new Lock(pool, issuer, Type.MAINTENANCE);
+ }
+
+ @Override
+ public Lock createRecoveryLock() {
+ return new Lock(pool, issuer, Type.RECOVERY);
+ }
+
+ @Override
+ public void close() throws IOException {
+ pool.close();
+ }
+
+ private void initializeLockTables() {
+ LOG.trace("Creating database tables (if missing) to store table maintenance locks");
+ try {
+ pool.run(
+ conn -> {
+ DatabaseMetaData dbMeta = conn.getMetaData();
+ ResultSet tableExists =
+ dbMeta.getTables(
+ null /* catalog name */,
+ null /* schemaPattern */,
+ LOCK_TABLE_NAME /* tableNamePattern */,
+ null /* types */);
+ if (tableExists.next()) {
+ return true;
+ }
+
+ LOG.debug("Creating table {} to store iceberg catalog tables", LOCK_TABLE_NAME);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1636082614
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
Review Comment:
This is the `TableChange` per scheduled Maintenance Task since the last Trigger of the given task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1636079471
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
Review Comment:
State handling and timers need keyed operators... This is awkward, but we haven't find a better solution.
Maybe another idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1637197094
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 3), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 10), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 0);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 1, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(5, 7, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 1, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 3, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 5, 7, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 1, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(3, 1, 0, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 2, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 10, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = new TableChange(1, 0, 0, 0, 1);
+
+ // Wait for one trigger
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for a second trigger
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(new TableChange(1, 0, 0, 0, 1), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 1), EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues())
+ .isEqualTo(Lists.newArrayList(Trigger.cleanUp(testHarness.getProcessingTime())));
+
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testCleanup(boolean cleanup) throws Exception {
+ // Simulate a lock remaining from a previous run
+ assertThat(lock.tryLock()).isTrue();
+
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, 1, cleanup);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ assertThat(lock.isHeld()).isEqualTo(!cleanup);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.cleanUp(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L,
+ false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private static Stream<Arguments> parametersForTestRecovery() {
+ return Stream.of(
+ Arguments.of(true, false),
+ Arguments.of(true, false),
+ Arguments.of(false, true),
+ Arguments.of(false, false));
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED,
+ rateLimiterTrigger)
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED,
+ concurrentRunTrigger)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ lock.unlock();
+ }
+
+ private static TriggerManager manager(
+ TableLoader tableLoader,
+ TriggerEvaluator evaluator,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ return new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1),
+ Lists.newArrayList(evaluator),
+ minFireDelayMs,
+ lockCheckDelayMs,
+ clearLocks);
+ }
+
+ private static TriggerEvaluator triggerEvaluatorWith2Commits() {
+ return new TriggerEvaluator.Builder().commitNumber(2).build();
+ }
+
+ private static TriggerManager manager(TableLoader tableLoader, TriggerEvaluator evaluator) {
+ return manager(tableLoader, evaluator, 1, 1, false);
+ }
+
+ private static TriggerManager manager(TableLoader tableLoader) {
Review Comment:
not sure if this hiding helps with code understanding. it is easier to understand if the evaluator is explicitly defined
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
Review Comment:
how is the `clearLocks` set by the caller? or what scenario it should be true?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
Review Comment:
circling back on the `TableChange` interface from last PR. this style of constructor is bit hard to remember what each of those 5 numbers are for. a builder pattern may be a little easier. in this case, it could be `TableChange.builder().commitNumber(1).build`
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 3), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 10), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 0);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 1, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(5, 7, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 1, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 3, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 5, 7, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 1, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(3, 1, 0, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 2, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 10, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = new TableChange(1, 0, 0, 0, 1);
+
+ // Wait for one trigger
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for a second trigger
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(new TableChange(1, 0, 0, 0, 1), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 1), EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues())
+ .isEqualTo(Lists.newArrayList(Trigger.cleanUp(testHarness.getProcessingTime())));
+
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testCleanup(boolean cleanup) throws Exception {
+ // Simulate a lock remaining from a previous run
+ assertThat(lock.tryLock()).isTrue();
+
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, 1, cleanup);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ assertThat(lock.isHeld()).isEqualTo(!cleanup);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.cleanUp(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L,
+ false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private static Stream<Arguments> parametersForTestRecovery() {
+ return Stream.of(
+ Arguments.of(true, false),
+ Arguments.of(true, false),
+ Arguments.of(false, true),
+ Arguments.of(false, false));
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED,
+ rateLimiterTrigger)
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED,
+ concurrentRunTrigger)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ lock.unlock();
Review Comment:
why unlock for each of this method? where is it locked? ideally, it is easier to read if lock and unlock happens in the same scope/method.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
Review Comment:
this is essentially a delayed cleanup, because timer service is not available in this context?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
Review Comment:
why don't we put name inside the `TriggerEvaluator` class?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
Review Comment:
what if lock was created/acquired by another process?
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 3), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 10), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 0);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 1, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(5, 7, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 1, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 3, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 5, 7, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 1, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(3, 1, 0, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 2, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 10, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = new TableChange(1, 0, 0, 0, 1);
+
+ // Wait for one trigger
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for a second trigger
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(new TableChange(1, 0, 0, 0, 1), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 1), EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues())
+ .isEqualTo(Lists.newArrayList(Trigger.cleanUp(testHarness.getProcessingTime())));
+
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testCleanup(boolean cleanup) throws Exception {
+ // Simulate a lock remaining from a previous run
+ assertThat(lock.tryLock()).isTrue();
+
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, 1, cleanup);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ assertThat(lock.isHeld()).isEqualTo(!cleanup);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.cleanUp(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L,
+ false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private static Stream<Arguments> parametersForTestRecovery() {
+ return Stream.of(
+ Arguments.of(true, false),
+ Arguments.of(true, false),
+ Arguments.of(false, true),
+ Arguments.of(false, false));
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED,
+ rateLimiterTrigger)
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED,
+ concurrentRunTrigger)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ lock.unlock();
+ }
+
+ private static TriggerManager manager(
+ TableLoader tableLoader,
+ TriggerEvaluator evaluator,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ return new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1),
+ Lists.newArrayList(evaluator),
+ minFireDelayMs,
+ lockCheckDelayMs,
+ clearLocks);
+ }
+
+ private static TriggerEvaluator triggerEvaluatorWith2Commits() {
Review Comment:
nit: is this helper method beneficial? I would think the original form is a bit easier to read/understand
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService(), timestamp);
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime.value());
+ }
+
+ private void schedule(TimerService timerService, long time) throws IOException {
+ nextEvaluationTime.update(time);
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService, long current)
+ throws Exception {
+ if (!inited) {
+ // Initialize with empty changes and current timestamp
+ if (!accumulatedChanges.get().iterator().hasNext()) {
+ List<TableChange> changes = Lists.newArrayListWithCapacity(evaluators.size());
+ List<Long> triggerTimes = Lists.newArrayListWithCapacity(evaluators.size());
+ for (int i = 0; i < evaluators.size(); ++i) {
+ changes.add(TableChange.empty());
+ triggerTimes.add(current);
+ }
+
+ accumulatedChanges.update(changes);
+ lastTriggerTimes.update(triggerTimes);
+ }
+
+ if (isCleanUp) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a cleanup lock
+ // - fire a clean-up trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end.
+ recoveryLock.tryLock();
Review Comment:
where is recoveryLock unlocked?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1637602939
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
Review Comment:
I've tried this and failed for some reason. Let me double check this and come back with the results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639280951
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
Review Comment:
These are the specific Maintenance Tasks scheduled by the user. It could be one or more of the following:
- DataFileRewrite
- ManifestFileRewrite
- ExpireSnapshots
- DeleteOrphanFiles
We can add more later, if we need/want.
Also, the user might decide to schedule a given type multiple times with different configurations, like RewriteDataFiles with only small file compaction on every 10 commits, but a full table rewrite on every day.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701478003
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
Review Comment:
Good catch!
Not needed.
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699651831
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(Table table, String lockKey) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockKey, "Lock key should not be null");
+ this.table = table;
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
Review Comment:
> wondering if retry is safe here. Could the lock/unblock action initiated by multiple threads/operator subtasks?
This is a valid issue. We need to write a more complicated locking with lockIds, and checks like we have in `MetastoreLock` and `LockRequest`. The main idea is to have something like `__flink_maitenance_<RND>` for the lock.
- For checking the lock, we check if there is any tag starting with `__flink_maitenance_`
- For creating the lock, we create a tag with a `__flink_maitenance_<RND>`
- For retries for creation, we check if the lock with `RND` is created, and if not, try to create again
- For removal, we list the locks and remove the tag with `__flink_maitenance_<RND>`
- For retries for removal, we check if the lock with the `RND` is removed, and if not, try to remove again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1700306350
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ private static final String RUNNING_TAG = "__flink_maintenance_running";
+ private static final String RECOVERING_TAG = "__flink_maintenance_recovering";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockKey;
+
+ public Lock(Table table, String lockKey) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockKey, "Lock key should not be null");
+ this.table = table;
+ this.lockKey = lockKey;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held");
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
Review Comment:
> > also how do we deal with orphaned locks where unlock failed (e.g. due to network issue with catalog service)
>
> We could reuse the recovery lock mechanism for this situation: After a given period, if we can't acquire the lock, then we can create a recovery lock, and send a `Trigger.cleanup`. If the recovery lock is not removed after a time period we could just throw an exception.
The `unlock` will retry, and if it is failed `CHANGE_ATTEMPTS` times, we throw an exception. If we face many user issues where the retries and the job failure is not enough, we can examine the situations and improve the stability further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1699585173
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
Review Comment:
We need changes in `TriggerManager` (handling cleanup) and the `Lock` (handling lock priority) implementation/requirements as well to handle multiple Flink jobs running compaction.
Based on this I added the comment to `TriggerManager` instead:
```
The current implementation only handles conflicts within a single job. Users should avoid scheduling
maintenance for the same table in different Flink jobs.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701520536
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
Review Comment:
In the `TriggerManager.init` method:
```
if (nextEvaluationTime == null) {
schedule(timerService, current + minFireDelayMs);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1702043002
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TagBasedLockFactory.class);
+ @VisibleForTesting static final String RUNNING_TAG_PREFIX = "__flink_maintenance_running_";
+ @VisibleForTesting static final String RECOVERING_TAG_PREFIX = "__flink_maintenance_recovering_";
+ private static final int CHANGE_ATTEMPTS = 3;
+
+ private final TableLoader tableLoader;
+ private transient Table table;
+
+ public TagBasedLockFactory(TableLoader tableLoader) {
+ this.tableLoader = tableLoader;
+ }
+
+ @Override
+ public void open() {
+ tableLoader.open();
+ this.table = tableLoader.loadTable();
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createLock() {
+ return new Lock(table, RUNNING_TAG_PREFIX);
+ }
+
+ @Override
+ public TriggerLockFactory.Lock createRecoveryLock() {
+ return new Lock(table, RECOVERING_TAG_PREFIX);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ }
+
+ public static class Lock implements TriggerLockFactory.Lock {
+ private final Table table;
+ private final String lockPrefix;
+
+ public Lock(Table table, String lockPrefix) {
+ Preconditions.checkNotNull(table, "Table should not be null");
+ Preconditions.checkNotNull(lockPrefix, "Lock key should not be null");
+ this.table = table;
+ this.lockPrefix = lockPrefix;
+ }
+
+ /**
+ * The lock will be acquired by jobs with creating a new tag. A new empty commit is added for a
+ * table without snapshots.
+ *
+ * @return <code>true</code> if the lock is acquired by this operator
+ */
+ @Override
+ public boolean tryLock() {
+ if (isHeld()) {
+ LOG.info("Lock is already held. The relevant tag: {}", findLock());
+ return false;
+ }
+
+ if (table.currentSnapshot() == null) {
+ // Create an empty commit
+ table.newFastAppend().commit();
+ LOG.info("Empty table, new empty commit added for using tags");
+ }
+
+ String lockKey = lockPrefix + UUID.randomUUID();
+ try {
+ Tasks.foreach(1)
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ unused -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.createTag(lockKey, table.currentSnapshot().snapshotId());
+ manage.commit();
+ LOG.debug("Lock created");
+ } catch (Exception e) {
+ if (!isHeld()) {
+ LOG.warn("Retrying lock after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock created, hiding exception", e);
+ }
+ }
+ });
+ } catch (Exception e) {
+ LOG.info("Concurrent lock created. Is there a concurrent maintenance job running?", e);
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ public void unlock() {
+ table.refresh();
+
+ Optional<String> lockKey = findLock();
+ if (lockKey.isPresent()) {
+ Tasks.foreach(lockKey.get())
+ .retry(CHANGE_ATTEMPTS)
+ .stopOnFailure()
+ .throwFailureWhenFinished()
+ .run(
+ key -> {
+ try {
+ table.refresh();
+ ManageSnapshots manage = table.manageSnapshots();
+ manage.removeTag(key);
+ manage.commit();
+ } catch (Exception e) {
+ table.refresh();
+ if (table.refs().containsKey(key)) {
+ LOG.warn("Retrying lock removal after exception", e);
+ throw e;
+ } else {
+ LOG.debug("Lock removed, hiding exception", e);
+ }
+ }
+ });
+ LOG.debug("Lock removed");
+ } else {
+ LOG.warn(
+ "Missing lock, can not remove. Found only the following tags: {}.",
+ table.refs().keySet());
+ }
+ }
+
+ @Override
+ public boolean isHeld() {
+ table.refresh();
+ return findLock().isPresent();
+ }
+
+ private Optional<String> findLock() {
+ List<String> locks =
+ table.refs().keySet().stream()
+ .filter(tag -> tag.startsWith(lockPrefix))
+ .collect(Collectors.toList());
+ Preconditions.checkArgument(locks.size() < 2, "Invalid lock state: %s", locks);
Review Comment:
nit: should explain what is invalid here. maybe sth like `Invalid lock state: multiple locks found: %s`
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
Review Comment:
oh. should this line of `processElement` be called right after line 186? check there is no output immediately after the element and one output after timeout (clock advanced)?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
Review Comment:
adding the check in `TriggerManager.init` won't help. it happens first. we should add the `if (nextEvaluationTime == null)` check before every `schedule` call. in particularly, 2 and 3 in `checkAndFire` method
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
Review Comment:
oh. I missed one TableChange event. thought there was only 3 commits.
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTagBasedLockFactory.java:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TagBasedLockFactory.RUNNING_TAG_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.CommitFailedException;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTagBasedLockFactory extends TestLockFactoryBase {
Review Comment:
> reusing the OperatorTestBase
It would just lead me to wonder why is operator needed to test lock factory. It is not a blocker from my side if you have clear preference of this pattern.
> TableLoader is needed for the TagBasedLockFactory.
maybe use existing `HadoopTableExtension` which extends `HadoopCatalogExtension`?
> Also SQL is used to create the table, and add a snapshot to it.
maybe `GenericAppenderHelper` from iceberg-core?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean shouldRestoreTasks = false;
+ private transient boolean inited = false;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ shouldRestoreTasks = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ @Override
+ public void close() throws IOException {
+ tableLoader.close();
+ lockFactory.close();
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (shouldRestoreTasks) {
+ if (recoveryLock.isHeld()) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ shouldRestoreTasks = false;
+ }
+ }
+
+ Integer taskToStart =
+ nextTrigger(evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = accumulatedChanges.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime);
+ }
+
+ private void schedule(TimerService timerService, long time) {
+ this.nextEvaluationTime = time;
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService) throws Exception {
+ if (!inited) {
+ long current = timerService.currentProcessingTime();
+
+ // Initialize from state
+ this.nextEvaluationTime = nextEvaluationTimeState.value();
Review Comment:
keyed state only available after `keyBy`. doubt it is related to construct the state variable in the `open` method or `initializeState` method.
This should probably also be an operator list state with a single element. This way, we don't need the keyed stream. I remember we discussed the keyed stream before. is this the reason/problem?
```
this.nextEvaluationTimeState =
getRuntimeContext()
.getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
```
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/MetricsReporterFactoryForTests.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.MetricReporterFactory;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+
+public class MetricsReporterFactoryForTests implements MetricReporterFactory {
Review Comment:
not sure if I understand the question. I was asking if we can replace this class with Flink's existing `CollectingMetricsReporter`. Its base class stores the metrics using Map like here.
```
public abstract class AbstractReporter implements MetricReporter, CharacterFilter {
protected final Logger log = LoggerFactory.getLogger(getClass());
protected final Map<Gauge<?>, String> gauges = new HashMap<>();
protected final Map<Counter, String> counters = new HashMap<>();
protected final Map<Histogram, String> histograms = new HashMap<>();
protected final Map<Meter, String> meters = new HashMap<>();
```
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
Review Comment:
I am saying if `concurrentRunTriggered` is the accurate metric name. seems more like `concurrentRunThrottled` to me, because it is not triggered and was scheduled for later check instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701471050
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
Review Comment:
Added:
```
// Simulates that the recovered maintenance task finishes and removes the lock
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701466221
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1701493223
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory lockFactory;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ this.lockFactory = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME));
+ lockFactory.open();
+ this.lock = lockFactory.createLock();
+ this.recoveringLock = lockFactory.createRecoveryLock();
+ lock.unlock();
+ recoveringLock.unlock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ lockFactory.close();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 0);
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(1).deleteFileNum(1).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(5).deleteFileNum(7).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 0);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(1L).deleteFileSize(1L).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(3L).build(), 2);
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileSize(5L).deleteFileSize(7L).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileSize(1L).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().dataFileSize(1L).build(), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ testHarness, TableChange.builder().dataFileNum(3).deleteFileNum(1).build(), 0);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(2).build(), 1);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(3).build(), 2);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 3);
+
+ addEventAndCheckResult(testHarness, TableChange.builder().deleteFileNum(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = TableChange.builder().dataFileSize(1).commitNum(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(
+ TableChange.builder().dataFileSize(1).commitNum(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(TableChange.builder().commitNum(1).build(), EVENT_TIME_2);
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ // Remove the lock to allow the next trigger
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, TableChange.builder().commitNum(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.recovery(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ assertTriggers(testHarness.extractOutputValues(), expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(TableChange.builder().commitNum(2).build(), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertTriggers(testHarness.extractOutputValues(), expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ lockFactory,
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitNum(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
Review Comment:
This is also testing that there are no other "random" counters there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1648221004
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JVMBasedLockFactory.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.concurrent.Semaphore;
+import org.apache.flink.annotation.Internal;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The locks are based on static {@link Semaphore} objects. We expect that the {@link
+ * TriggerManager} and the LockRemover operators will be placed on the same TaskManager (JVM), as
+ * they are both global. In this case JVM based locking should be enough to allow communication
+ * between the operators.
+ */
+@Internal
+public class JVMBasedLockFactory implements TriggerLockFactory {
Review Comment:
nit: JVM seems redundant as this is Java code. maybe `SemaphoreLockFactory` or `InMemorySemaphoreLockFactory`?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
Review Comment:
nit: `shouldCleanUp` seems more accurate. `isCleanUp` makes more sense in the `Trigger` class
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
Review Comment:
do we need to check if `nextTime` is larger than the `current` time here?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
Review Comment:
nit: It is probably more common to keep these lists as class members. and in the `snapshotState` method, just update the state variable with the list.
for this use case (low-frequency state read/write), it doesn't matter performance wise (either way).
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
should we avoid `TagBasedLockFactory` to start with? I am still a bit concerned about using Tag as distributed lock mechanism just semantically.
`SemaphoreLock` is a simple and good implementation to provide. It does has some limitations. E.g., we probably can't specify the slot sharing group to control the colocation for Flink SQL jobs?
Another potential implementation is using k8s ConfigMap/Etcd. but we probably don't want to pull in the k8s Java SDK dep in Flink module.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
Review Comment:
I am not sure if we are talking about the same thing. I was saying moving the `String taskName` into the `TriggerEvaluator` class. this way, we only need `List<TriggerEvaluator> evaluators`. Then we don't need the check on `taskNames.size() == evaluators.size()`
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
Review Comment:
nit: add the time unit `1 ms`. same for the error msg below
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerEvaluator.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.List;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Internal
+class TriggerEvaluator implements Serializable {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerEvaluator.class);
+ private final List<Predicate> predicates;
+
+ private TriggerEvaluator(List<Predicate> predicates) {
+ Preconditions.checkArgument(!predicates.isEmpty(), "Provide at least 1 condition.");
+
+ this.predicates = predicates;
+ }
+
+ boolean check(TableChange event, long lastTimeMs, long currentTimeMs) {
+ boolean result =
+ predicates.stream()
+ .anyMatch(
+ p -> {
+ try {
+ return p.evaluate(event, lastTimeMs, currentTimeMs);
+ } catch (Exception e) {
+ throw new RuntimeException("Error accessing state", e);
+ }
+ });
+ LOG.debug(
+ "Checking event: {}, at {}, last: {} with result: {}",
+ event,
+ currentTimeMs,
+ lastTimeMs,
+ result);
+ return result;
+ }
+
+ static class Builder implements Serializable {
+ private Integer commitNumber;
+ private Integer fileNumber;
+ private Long fileSize;
+ private Integer deleteFileNumber;
Review Comment:
do we need `deleteFileBytes`?
I am also wondering if we need to separate out position delete and equality deletes for trigger evaluator.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
Review Comment:
nit: move the variable definitions right before the usage for readability
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/ConstantsForTests.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+class ConstantsForTests {
+ public static final long EVENT_TIME = 10L;
Review Comment:
I am not sure these constants improve readability. e.g., what is the relationship (comparison) btw EVENT_TIME and EVENT_TIME_2? it would be easier to read the literals directly.
similar comment for dummy name.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
Review Comment:
thanks for verifying. please add the context to the class javadoc, which is empty currently.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
Review Comment:
oh. I got it now. the list is has one `TableChange` accumulation for each maintenance task
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerEvaluator.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.List;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Internal
+class TriggerEvaluator implements Serializable {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerEvaluator.class);
+ private final List<Predicate> predicates;
+
+ private TriggerEvaluator(List<Predicate> predicates) {
+ Preconditions.checkArgument(!predicates.isEmpty(), "Provide at least 1 condition.");
+
+ this.predicates = predicates;
+ }
+
+ boolean check(TableChange event, long lastTimeMs, long currentTimeMs) {
+ boolean result =
+ predicates.stream()
+ .anyMatch(
+ p -> {
+ try {
+ return p.evaluate(event, lastTimeMs, currentTimeMs);
+ } catch (Exception e) {
+ throw new RuntimeException("Error accessing state", e);
+ }
+ });
+ LOG.debug(
+ "Checking event: {}, at {}, last: {} with result: {}",
+ event,
+ currentTimeMs,
+ lastTimeMs,
+ result);
+ return result;
+ }
+
+ static class Builder implements Serializable {
+ private Integer commitNumber;
+ private Integer fileNumber;
Review Comment:
nit: can we use variable naming pattern similar to existing code? Here are some from `PartitionsTable` class.
```
private int dataFileCount;
private long dataFileSizeInBytes;
private long posDeleteRecordCount;
private int posDeleteFileCount;
private long eqDeleteRecordCount;
private int eqDeleteFileCount;
```
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
Review Comment:
in case of nothing to trigger, why do we need to reset the cursor/position to 0?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JVMBasedLockFactory.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.concurrent.Semaphore;
+import org.apache.flink.annotation.Internal;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The locks are based on static {@link Semaphore} objects. We expect that the {@link
+ * TriggerManager} and the LockRemover operators will be placed on the same TaskManager (JVM), as
+ * they are both global. In this case JVM based locking should be enough to allow communication
Review Comment:
just global operator doesn't guarantee the colocation. we probably should call out the Flink slot sharing group. some brief example would be helpful.
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
Review Comment:
for each `Trigger` record, we are loading the table from catalog. why do we let downstream operator load the table directly? i.e. downstream operator gets the `TableLoader` and we avoid including `SerializableTable` to the `Trigger` object?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
Review Comment:
might be a bit more readable to do the `mod` here (than in the `nextTrigger` method)
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService(), timestamp);
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime.value());
+ }
+
+ private void schedule(TimerService timerService, long time) throws IOException {
+ nextEvaluationTime.update(time);
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService, long current)
+ throws Exception {
+ if (!inited) {
+ // Initialize with empty changes and current timestamp
+ if (!accumulatedChanges.get().iterator().hasNext()) {
+ List<TableChange> changes = Lists.newArrayListWithCapacity(evaluators.size());
+ List<Long> triggerTimes = Lists.newArrayListWithCapacity(evaluators.size());
+ for (int i = 0; i < evaluators.size(); ++i) {
+ changes.add(TableChange.empty());
+ triggerTimes.add(current);
+ }
+
+ accumulatedChanges.update(changes);
+ lastTriggerTimes.update(triggerTimes);
+ }
+
+ if (isCleanUp) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a cleanup lock
+ // - fire a clean-up trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end.
+ recoveryLock.tryLock();
Review Comment:
we don't check the boolean return value from `tryLock`. is that correct?
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
Review Comment:
nit: maybe add the context as code comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1651100591
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
Review Comment:
for each `Trigger` record, we are loading the table from catalog. why don't we let downstream operator load the table directly? i.e. downstream operator gets the `TableLoader` and we avoid including `SerializableTable` to the `Trigger` object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698418626
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManager.java:
##########
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.DUMMY_NAME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME;
+import static org.apache.iceberg.flink.maintenance.operator.ConstantsForTests.EVENT_TIME_2;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.CONCURRENT_RUN_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.GROUP_VALUE_DEFAULT;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.MetricConstants.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.stream.Stream;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestTriggerManager extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String NAME_1 = "name1";
+ private static final String NAME_2 = "name2";
+ private long processingTime = 0L;
+ private TriggerLockFactory.Lock lock;
+ private TriggerLockFactory.Lock recoveringLock;
+
+ @BeforeEach
+ void before() {
+ sql.exec("CREATE TABLE %s (id int, data varchar)", TABLE_NAME);
+ lock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createLock();
+ recoveringLock = new TagBasedLockFactory(sql.tableLoader(TABLE_NAME)).createRecoveryLock();
+ MetricsReporterFactoryForTests.reset();
+ }
+
+ @Test
+ void testCommitNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().commitNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 3), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 10), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 1), 4);
+ }
+ }
+
+ @Test
+ void testFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 0);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 1, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(5, 7, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(1, 0, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testFileSize() throws Exception {
+ TriggerManager manager =
+ manager(sql.tableLoader(TABLE_NAME), new TriggerEvaluator.Builder().fileSize(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 1, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 3, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 5, 7, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 1, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 1, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testDeleteFileNumber() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().deleteFileNumber(3).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(3, 1, 0, 0, 0), 0);
+ addEventAndCheckResult(testHarness, new TableChange(0, 2, 0, 0, 0), 1);
+ addEventAndCheckResult(testHarness, new TableChange(0, 3, 0, 0, 0), 2);
+ addEventAndCheckResult(testHarness, new TableChange(0, 10, 0, 0, 0), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 3);
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 1, 0, 0, 0), 4);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManager manager =
+ manager(
+ sql.tableLoader(TABLE_NAME),
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build());
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ TableChange event = new TableChange(1, 0, 0, 0, 1);
+
+ // Wait for one trigger
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for a second trigger
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ testHarness.processElement(new TableChange(1, 0, 0, 0, 1), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ manager = manager(tableLoader);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Arrives the first real change
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 1), EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues())
+ .isEqualTo(Lists.newArrayList(Trigger.cleanUp(testHarness.getProcessingTime())));
+
+ recoveringLock.unlock();
+ testHarness.setProcessingTime(EVENT_TIME_2);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, DELAY, 1, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLogCheckDelay() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, DELAY, false);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+
+ // Create a lock to prevent execution, and check that there is no result
+ assertThat(lock.tryLock()).isTrue();
+ addEventAndCheckResult(testHarness, new TableChange(0, 0, 0, 0, 2), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ lock.unlock();
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testCleanup(boolean cleanup) throws Exception {
+ // Simulate a lock remaining from a previous run
+ assertThat(lock.tryLock()).isTrue();
+
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader, 1, 1, cleanup);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+
+ assertThat(lock.isHeld()).isEqualTo(!cleanup);
+ }
+ }
+
+ /**
+ * Simulating recovery scenarios where there is a leftover table lock, and ongoing maintenance
+ * task.
+ *
+ * @param tableLock left on the table on job recovery
+ * @param runningTask is running and continues to run after job recovery
+ */
+ @ParameterizedTest
+ @MethodSource("parametersForTestRecovery")
+ void testRecovery(boolean tableLock, boolean runningTask) throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+ TriggerManager manager = manager(tableLoader);
+ OperatorSubtaskState state;
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.open();
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ if (tableLock) {
+ assertThat(lock.tryLock()).isTrue();
+ }
+
+ manager = manager(tableLoader);
+ List<Trigger> expected = Lists.newArrayListWithExpectedSize(3);
+ try (KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> testHarness =
+ harness(manager)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ ++processingTime;
+ expected.add(Trigger.cleanUp(processingTime));
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // Nothing happens until the recovery is finished
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ if (runningTask) {
+ lock.unlock();
+ }
+
+ // Still no results as the recovery is ongoing
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+
+ // All locks are removed when the recovery trigger is received by the lock cleaner
+ lock.unlock();
+ recoveringLock.unlock();
+
+ // Emit only a single trigger
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(new TableChange(0, 0, 0, 0, 2), processingTime);
+ // Releasing lock will create a new snapshot, and we receive this in the trigger
+ expected.add(
+ Trigger.create(
+ processingTime,
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable()),
+ 0));
+ assertThat(testHarness.extractOutputValues()).isEqualTo(expected);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManager manager =
+ new TriggerManager(
+ tableLoader,
+ new TagBasedLockFactory(tableLoader),
+ Lists.newArrayList(NAME_1, NAME_2),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitNumber(2).build(),
+ new TriggerEvaluator.Builder().commitNumber(4).build()),
+ 1L,
+ 1L,
+ false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER);
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 1));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(1L);
+ lock.unlock();
+
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED, -1L)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 2L)
+ .put(DUMMY_NAME + "." + NAME_2 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1_000_000L, 1L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ lock.unlock();
+
+ // The second trigger will be blocked
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ TableLoader tableLoader = sql.tableLoader(TABLE_NAME);
+
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManager manager = manager(tableLoader, 1L, 1_000_000L, false);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .process(manager)
+ .name(DUMMY_NAME)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(new TableChange(0, 0, 0L, 0L, 2));
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED)
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private static Stream<Arguments> parametersForTestRecovery() {
+ return Stream.of(
+ Arguments.of(true, false),
+ Arguments.of(true, false),
+ Arguments.of(false, true),
+ Arguments.of(false, false));
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<String, Long>()
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + RATE_LIMITER_TRIGGERED,
+ rateLimiterTrigger)
+ .put(
+ DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + CONCURRENT_RUN_TRIGGERED,
+ concurrentRunTrigger)
+ .put(DUMMY_NAME + "." + NAME_1 + "." + TRIGGERED, 1L)
+ .put(DUMMY_NAME + "." + GROUP_VALUE_DEFAULT + "." + NOTHING_TO_TRIGGER, 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ lock.unlock();
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698428895
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
+ accumulated.forEach(tableChange -> tableChange.merge(change));
+ accumulatedChanges.update(accumulated);
+
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulated);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService(), timestamp);
+ nextEvaluationTime.clear();
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out)
+ throws Exception {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ } else {
+ LOG.info("The cleanup is finished at {}", current);
+ isCleanUp = false;
+ }
+ }
+
+ List<TableChange> changes = Lists.newArrayList(accumulatedChanges.get());
+ List<Long> times = Lists.newArrayList(lastTriggerTimes.get());
+ Integer taskToStart = nextTrigger(evaluators, changes, times, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (startsFrom == 0) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, changes);
+ }
+
+ startsFrom = 0;
+ return;
+ }
+
+ if (lock.tryLock()) {
+ TableChange change = changes.get(taskToStart);
+ SerializableTable table =
+ (SerializableTable) SerializableTable.copyOf(tableLoader.loadTable());
+ out.collect(Trigger.create(current, table, taskToStart));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, table.name());
+ triggerCounters.get(taskToStart).inc();
+ changes.set(taskToStart, TableChange.empty());
+ accumulatedChanges.update(changes);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = taskToStart + 1;
+ } else {
+ // The lock is already held by someone
+ LOG.info("Delaying task on failed lock check: {}", current);
+
+ startsFrom = taskToStart;
+ concurrentRunTriggeredCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+
+ timerService.registerProcessingTimeTimer(nextEvaluationTime.value());
+ }
+
+ private void schedule(TimerService timerService, long time) throws IOException {
+ nextEvaluationTime.update(time);
+ timerService.registerProcessingTimeTimer(time);
+ }
+
+ private static Integer nextTrigger(
+ List<TriggerEvaluator> evaluators,
+ List<TableChange> changes,
+ List<Long> lastTriggerTimes,
+ long currentTime,
+ int startPos) {
+ int normalizedStartingPos = startPos % evaluators.size();
+ int current = normalizedStartingPos;
+ do {
+ if (evaluators
+ .get(current)
+ .check(changes.get(current), lastTriggerTimes.get(current), currentTime)) {
+ return current;
+ }
+
+ current = (current + 1) % evaluators.size();
+ } while (current != normalizedStartingPos);
+
+ return null;
+ }
+
+ private void init(Collector<Trigger> out, TimerService timerService, long current)
+ throws Exception {
+ if (!inited) {
+ // Initialize with empty changes and current timestamp
+ if (!accumulatedChanges.get().iterator().hasNext()) {
+ List<TableChange> changes = Lists.newArrayListWithCapacity(evaluators.size());
+ List<Long> triggerTimes = Lists.newArrayListWithCapacity(evaluators.size());
+ for (int i = 0; i < evaluators.size(); ++i) {
+ changes.add(TableChange.empty());
+ triggerTimes.add(current);
+ }
+
+ accumulatedChanges.update(changes);
+ lastTriggerTimes.update(triggerTimes);
+ }
+
+ if (isCleanUp) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a cleanup lock
+ // - fire a clean-up trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end.
+ recoveryLock.tryLock();
Review Comment:
Added a comment like this:
```
The result of the 'tryLock' is ignored as an already existing lock prevents collisions as well
```
> if tryLock failed, it is fine to send the Trigger.cleanUp event?
Consider the situation that we tried to recover a failed job, created the recovery lock, but the job failed again before finishing the lock cleanup. In this case if we try to recover the failed job again, then we will have a lock, but no `Trigger.cleanUp` in the pipeline, so we need to fire a `Trigger.cleanUp` event, as the `state` of the table could be different from the `state` of the Flink job we are recovering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698437502
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -83,7 +96,7 @@ class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
Preconditions.checkArgument(
- lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1698477231
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManager starts the Maintenance Tasks by emitting {@link Trigger} messages which are
+ * calculated based on the incoming {@link TableChange} messages. The TriggerManager keeps track of
+ * the changes since the last run of the Maintenance Tasks and triggers a new run based on the
+ * result of the {@link TriggerEvaluator}.
+ *
+ * <p>The TriggerManager prevents overlapping Maintenance Task runs using {@link
+ * org.apache.iceberg.flink.maintenance.operator.TriggerLockFactory.Lock}.
+ *
+ * <p>The TriggerManager should run as a global operator. {@link KeyedProcessFunction} is used, so
+ * the timer functions are available, but the key is not used.
+ */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Long nextEvaluationTime;
+ private transient List<TableChange> accumulatedChanges;
+ private transient List<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1ms.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(
+ TableMaintenanceMetrics.GROUP_KEY, TableMaintenanceMetrics.GROUP_VALUE_DEFAULT)
+ .counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(TableMaintenanceMetrics.GROUP_KEY, name)
+ .counter(TableMaintenanceMetrics.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTimeState =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChangesState =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimesState =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ tableLoader.open();
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ if (inited) {
+ // Only store state if initialized
+ nextEvaluationTimeState.update(nextEvaluationTime);
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ }
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ LOG.info("Initializing state restored: {}", context.isRestored());
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ init(out, ctx.timerService());
+
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime;
+ if (nextTime == null) {
+ checkAndFire(current, ctx.timerService(), out);
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {}",
+ current,
+ nextTime,
+ accumulatedChanges);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Trigger> out) throws Exception {
+ init(out, ctx.timerService());
+ this.nextEvaluationTime = null;
+ checkAndFire(ctx.timerService().currentProcessingTime(), ctx.timerService(), out);
+ }
+
+ private void checkAndFire(long current, TimerService timerService, Collector<Trigger> out) {
+ if (isCleanUp) {
+ if (recoveryLock.isHeld()) {
+ LOG.debug("The cleanup lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
Review Comment:
We come here only by 2 routes:
- `processElement` - in this case `nextTime`/`nextEvaluationTime` is null - no previous timer scheduled
- `onTimer` - in this case the previous timer has fired, so no previous timer scheduled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #10484:
URL: https://github.com/apache/iceberg/pull/10484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1724518086
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JdbcLockFactory.java:
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.sql.DatabaseMetaData;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTimeoutException;
+import java.sql.SQLTransientConnectionException;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.iceberg.jdbc.JdbcClientPool;
+import org.apache.iceberg.jdbc.UncheckedInterruptedException;
+import org.apache.iceberg.jdbc.UncheckedSQLException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class JdbcLockFactory implements TriggerLockFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcLockFactory.class);
+ private static final String INIT_LOCK_TABLES_PROPERTY = "maintenance.lock.jdbc.init-lock-tables";
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1636134996
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
Review Comment:
Any better ideas?
The method initializes the state if it is not a job restart, and starts the recovery procedure to clean up locks and prevent concurrent Maintenance Task runs on recovery.
I was struggling to find a good name, so I'm open to any suggestions 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1637612192
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java:
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.util.Collector;
+import org.apache.iceberg.SerializableTable;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** . */
+@Internal
+class TriggerManager extends KeyedProcessFunction<Boolean, TableChange, Trigger>
+ implements CheckpointedFunction {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManager.class);
+
+ private final TableLoader tableLoader;
+ private final TriggerLockFactory lockFactory;
+ private final List<String> taskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private final long minFireDelayMs;
+ private final long lockCheckDelayMs;
+ private final boolean clearLocks;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunTriggeredCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private transient ValueState<Long> nextEvaluationTime;
+ private transient ListState<TableChange> accumulatedChanges;
+ private transient ListState<Long> lastTriggerTimes;
+ private transient TriggerLockFactory.Lock lock;
+ private transient TriggerLockFactory.Lock recoveryLock;
+ private transient boolean isCleanUp = false;
+ private transient boolean inited = false;
+ private transient int startsFrom = 0;
+
+ TriggerManager(
+ TableLoader tableLoader,
+ TriggerLockFactory lockFactory,
+ List<String> taskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ boolean clearLocks) {
+ Preconditions.checkNotNull(tableLoader, "Table loader should no be null");
+ Preconditions.checkNotNull(lockFactory, "Lock factory should no be null");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Evaluators should not be empty");
+ Preconditions.checkArgument(
+ taskNames.size() == evaluators.size(), "Provide a name and evaluator for all of the tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1.");
+
+ this.tableLoader = tableLoader;
+ this.lockFactory = lockFactory;
+ this.taskNames = taskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.clearLocks = clearLocks;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ this.rateLimiterTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunTriggeredCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.CONCURRENT_RUN_TRIGGERED);
+ this.nothingToTriggerCounter =
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, MetricConstants.GROUP_VALUE_DEFAULT)
+ .counter(MetricConstants.NOTHING_TO_TRIGGER);
+ this.triggerCounters =
+ taskNames.stream()
+ .map(
+ name ->
+ getRuntimeContext()
+ .getMetricGroup()
+ .addGroup(MetricConstants.GROUP_KEY, name)
+ .counter(MetricConstants.TRIGGERED))
+ .collect(Collectors.toList());
+
+ this.nextEvaluationTime =
+ getRuntimeContext()
+ .getState(new ValueStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+ this.accumulatedChanges =
+ getRuntimeContext()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+ this.lastTriggerTimes =
+ getRuntimeContext()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ // Do nothing
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) {
+ LOG.info("Initializing state restored: {}, clearLocks: {}", context.isRestored(), clearLocks);
+ this.lock = lockFactory.createLock();
+ this.recoveryLock = lockFactory.createRecoveryLock();
+ if (context.isRestored()) {
+ isCleanUp = true;
+ } else if (clearLocks) {
+ // Remove old lock if we are not restoring the job
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void processElement(TableChange change, Context ctx, Collector<Trigger> out)
+ throws Exception {
+ long current = ctx.timerService().currentProcessingTime();
+ Long nextTime = nextEvaluationTime.value();
+ init(out, ctx.timerService(), current);
+
+ // Add the new changes to the already accumulated ones
+ List<TableChange> accumulated = Lists.newArrayList(accumulatedChanges.get());
Review Comment:
We're planning to have multiple Maintenance Tasks in the PostCommitTopology. But we are planning to have a single `TriggerManager`, so it is easier to prevent concurrent runs. This single `TriggerManager` needs to handle the scheduling of every Maintenance Task. Every Maintenance Task could have a different scheduling requirement, like DataFileCompaction on ever 10 commits, ExpireSnapshots on every hour, and DeleteOrphanFiles on every day. For this, we need to keep track the accumulated `TableChange`s for every Maintenance Task.
OTOH some users might want to schedule Maintenance Tasks in a group, like every 10 commits do a DataFileCompaction, then do a ManifestFileCompaction, then do an ExpireSnapshots. If all of them has independent scheduling, then the order is not guaranteed, and also there fight for the lock.
That's why I opted for the single TriggerManager approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Maintenance - TriggerManager [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10484:
URL: https://github.com/apache/iceberg/pull/10484#discussion_r1639292398
##########
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Map;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.ManageSnapshots;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Iceberg table {@link ManageSnapshots#createTag(String, long)}/{@link
+ * ManageSnapshots#removeTag(String)} based lock implementation for {@link TriggerLockFactory}.
+ */
+@Internal
+public class TagBasedLockFactory implements TriggerLockFactory {
Review Comment:
Another idea:
- Is there any guarantee in Flink that the operators of the same subtasks, in the same slot sharing group are run on the same JVM?
Based on this graph: https://nightlies.apache.org/flink/flink-docs-master/fig/slot_sharing.svg, which I have found on this page: https://nightlies.apache.org/flink/flink-docs-master/docs/concepts/flink-architecture/#task-slots-and-resources this seems to be the case.
If we can make sure that the `TriggerManager` and the `LockRemover` are in the same slot sharing group, and both are global and run in the slot 0, then we could just use a JVM level static Java lock for locking.
Seems very hacky, but might work.
@stevenzwu, @gyfora: Do you know if there are any guarantees around the scheduling of the slots, and how to detect TaskManager recovery?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org