You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Guosmilesmile (via GitHub)" <gi...@apache.org> on 2026/01/27 08:59:29 UTC
[PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Guosmilesmile opened a new pull request, #15151:
URL: https://github.com/apache/iceberg/pull/15151
This PR is a continuation of https://github.com/apache/iceberg/pull/15042.
In this approach, the lock is maintained inside `TriggerManagerOperator`, and `TableMaintenanceCoordinator` acts as a bridge. It holds a reference to `TriggerManagerOperator` in a static map, and uses that to receive events from `LockRemoverOperator` and release the lock held by `TriggerManagerOperator`.
Normal flow
1. `TriggerManagerOperator` decides whether to fire a trigger and, if so, acquires the lock internally and sends the trigger downstream.
2. After the task is finished, the trigger or the watermark reaches `LockRemoverOperator`.
3. `LockRemoverOperator` sends an event to `TableMaintenanceCoordinator`. `TableMaintenanceCoordinator` uses the reference stored in its static map to find the corresponding `TriggerManagerOperator` and sends a “release lock” event to it.
4. `TriggerManagerOperator` releases the lock.
Recovery flow
1. During `initializeState`, `TriggerManagerOperator` marks itself as “in recovery” and then sends a recovery trigger downstream. While the “in recovery” flag is true, other tasks are not allowed to operate on it.
2. When the recovery trigger or the watermark reaches `LockRemoverOperator`, it sends an event to `TableMaintenanceCoordinator` to release the lock.
3. `TriggerManagerOperator` releases the lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2828798551
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
Review Comment:
Is it a problem if this method is called in both Coordinator?
It would be nice to move this method to the `BaseCoordinator`, so `PENDING_RELEASE_EVENTS` and `LOCK_RELEASE_CONSUMERS` could be private there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#issuecomment-3896553861
https://github.com/apache/iceberg/actions/runs/21983224902/job/63510848094?pr=15151
```
TestFlinkTableSink > testIcebergSinkDifferentDAG() > catalogName=testhadoop_basenamespace, baseNamespace=l0.l1, format=PARQUET, isStreaming=false, useV2Sink=true FAILED
java.lang.RuntimeException: Failed to collect table result
at org.apache.iceberg.flink.TestBase.sql(TestBase.java:105)
at org.apache.iceberg.flink.TestFlinkTableSink.testIcebergSinkDifferentDAG(TestFlinkTableSink.java:304)
Caused by:
org.apache.flink.table.api.TableException: Failed to wait job finish
at app//org.apache.flink.table.api.internal.InsertResultProvider.hasNext(InsertResultProvider.java:85)
at app//org.apache.flink.table.api.internal.InsertResultProvider$Iterator.hasNext(InsertResultProvider.java:112)
at app//org.apache.iceberg.relocated.com.google.common.collect.Iterators.addAll(Iterators.java:369)
at app//org.apache.iceberg.relocated.com.google.common.collect.Lists.newArrayList(Lists.java:151)
at app//org.apache.iceberg.flink.TestBase.sql(TestBase.java:103)
... 1 more
Caused by:
java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at java.base/java.util.conc
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2817205016
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
Review Comment:
That's what I meant. We still need to synchronize on the same object, but we can make it method-level by enclosing the entire code in the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816212400
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinatorProvider.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator;
+
+@Internal
+public class TriggerManagerCoordinatorProvider extends RecreateOnResetOperatorCoordinator.Provider {
Review Comment:
Could we move this to the TriggerManagerOperatorFactory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821271611
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManagerOperator to TriggerManagerCoordinator to register a lock release
+ * handler. This handler will be used to forward lock release events back to the operator when
+ * triggered by downstream operators.
+ */
+public class LockRegisterEvent implements OperatorEvent {
+
+ private final String lockId;
+
+ public LockRegisterEvent(String lockId) {
+ this.lockId = lockId;
+ }
+
+ public String lockId() {
+ return lockId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
Review Comment:
Do we need the equals and the hash?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManagerOperator to TriggerManagerCoordinator to register a lock release
+ * handler. This handler will be used to forward lock release events back to the operator when
+ * triggered by downstream operators.
+ */
+public class LockRegisterEvent implements OperatorEvent {
+
+ private final String lockId;
+
+ public LockRegisterEvent(String lockId) {
+ this.lockId = lockId;
+ }
+
+ public String lockId() {
+ return lockId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
Review Comment:
Do we need the equals and the hashCode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821331800
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestTableMaintenanceCoordinationLock.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.api;
+
+import static org.apache.iceberg.flink.SimpleDataUtil.createRowData;
+import static org.apache.iceberg.flink.maintenance.api.TableMaintenance.SOURCE_OPERATOR_NAME_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.Collections;
+import java.util.List;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.transformations.SourceTransformation;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.flink.maintenance.operator.ManualSource;
+import org.apache.iceberg.flink.maintenance.operator.OperatorTestBase;
+import org.apache.iceberg.flink.maintenance.operator.TableChange;
+import org.apache.iceberg.flink.sink.FlinkSink;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+class TestTableMaintenanceCoordinationLock extends OperatorTestBase {
+ private static final String MAINTENANCE_TASK_NAME = "TestTableMaintenance";
+ private static final String[] TASKS =
+ new String[] {MAINTENANCE_TASK_NAME + " [0]", MAINTENANCE_TASK_NAME + " [1]"};
+ private static final TableChange DUMMY_CHANGE = TableChange.builder().commitCount(1).build();
+ private static final List<Trigger> PROCESSED =
+ Collections.synchronizedList(Lists.newArrayListWithCapacity(1));
+
+ private StreamExecutionEnvironment env;
+ private Table table;
+
+ @TempDir private File checkpointDir;
+
+ @BeforeEach
+ public void beforeEach() throws IOException {
+ Configuration config = new Configuration();
+ config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
+ config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "file://" + checkpointDir.getPath());
+ this.env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ this.table = createTable();
+ insert(table, 1, "a");
+
+ PROCESSED.clear();
+ MaintenanceTaskBuilderForTest.counter = 0;
+ }
+
+ @Test
+ void testForChangeStream() throws Exception {
+ ManualSource<TableChange> schedulerSource =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+
+ TableMaintenance.Builder streamBuilder =
+ TableMaintenance.forChangeStream(schedulerSource.dataStream(), tableLoader())
+ .rateLimit(Duration.ofMillis(2))
+ .lockCheckDelay(Duration.ofSeconds(3))
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .scheduleOnDataFileCount(2)
+ .scheduleOnDataFileSize(3L)
+ .scheduleOnEqDeleteFileCount(4)
+ .scheduleOnEqDeleteRecordCount(5L)
+ .scheduleOnPosDeleteFileCount(6)
+ .scheduleOnPosDeleteRecordCount(7L)
+ .scheduleOnInterval(Duration.ofHours(1)));
+
+ sendEvents(schedulerSource, streamBuilder, ImmutableList.of(Tuple2.of(DUMMY_CHANGE, 1)));
+ }
+
+ @Test
+ void testForTable() throws Exception {
+ TableLoader tableLoader = tableLoader();
+
+ env.enableCheckpointing(10);
+
+ TableMaintenance.forTable(env, tableLoader)
+ .rateLimit(Duration.ofMillis(2))
+ .maxReadBack(2)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(2))
+ .append();
+
+ // Creating a stream for inserting data into the table concurrently
+ ManualSource<RowData> insertSource =
+ new ManualSource<>(env, InternalTypeInfo.of(FlinkSchemaUtil.convert(table.schema())));
+ FlinkSink.forRowData(insertSource.dataStream())
+ .tableLoader(tableLoader)
+ .uidPrefix(UID_SUFFIX + "-iceberg-sink")
+ .append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ insertSource.sendRecord(createRowData(2, "b"));
+
+ Awaitility.await().until(() -> PROCESSED.size() == 1);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testUidAndSlotSharingGroup() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupUnset() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, null);
+ checkSlotSharingGroupsAreSet(env, null);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupInherit() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupOverWrite() throws IOException {
+ String anotherUid = "Another-UID";
+ String anotherSlotSharingGroup = "Another-SlotSharingGroup";
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(anotherUid)
+ .slotSharingGroup(anotherSlotSharingGroup))
+ .append();
+
+ // Choose an operator from the scheduler part of the graph
+ Transformation<?> schedulerTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().equals("Trigger manager"))
+ .findFirst()
+ .orElseThrow();
+ assertThat(schedulerTransformation.getUid()).contains(UID_SUFFIX);
+ assertThat(schedulerTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(schedulerTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(SLOT_SHARING_GROUP);
+
+ // Choose an operator from the maintenance task part of the graph
+ Transformation<?> scheduledTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().startsWith(MAINTENANCE_TASK_NAME))
+ .findFirst()
+ .orElseThrow();
+ assertThat(scheduledTransformation.getUid()).contains(anotherUid);
+ assertThat(scheduledTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(scheduledTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(anotherSlotSharingGroup);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupForMonitorSource() throws IOException {
+ TableMaintenance.forTable(env, tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ Transformation<?> source = monitorSource();
+ assertThat(source).isNotNull();
+ assertThat(source.getUid()).contains(UID_SUFFIX);
+ assertThat(source.getSlotSharingGroup()).isPresent();
+ assertThat(source.getSlotSharingGroup().get().getName()).isEqualTo(SLOT_SHARING_GROUP);
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ /**
+ * Sends the events though the {@link ManualSource} provided, and waits until the given number of
+ * records are processed.
+ *
+ * @param schedulerSource used for sending the events
+ * @param streamBuilder used for generating the job
+ * @param eventsAndResultNumbers the pair of the event and the expected processed records
+ * @throws Exception if any
+ */
+ private void sendEvents(
Review Comment:
Is it worth to refactor and reuse `MaintenanceTaskTestBase.runAndWaitForSuccess`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2736185201
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
+ * otherwise, it schedules the next attempt. When the job recovers from a failure, tasks from
+ * different execution paths of the previous run may still be running. Therefore, it first needs to
+ * send a lock with the maximum timestamp, and then send a recovery trigger. Only after the
+ * downstream removes this lock can we be sure that all tasks have fully stopped.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient boolean restoreTasks = false;
+ private transient boolean lockHeld = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
Review Comment:
Here we are mainly registering something like a hook that can receive the “rock remove” signal from coordinate.
It is registered once when the job starts for the first time or when state is restored, and this registration overwrites the previous one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2731216535
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverOperator.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.iceberg.flink.maintenance.api.TaskResult;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class LockRemoverOperator extends AbstractStreamOperator<Void>
+ implements OneInputStreamOperator<TaskResult, Void>, OperatorEventHandler {
+ private static final Logger LOG = LoggerFactory.getLogger(LockRemoverOperator.class);
+
+ private static final long serialVersionUID = 1L;
+
+ private final String tableName;
+ private final OperatorEventGateway operatorEventGateway;
+ private final List<String> maintenanceTaskNames;
+ private transient List<Counter> succeededTaskResultCounters;
+ private transient List<Counter> failedTaskResultCounters;
+ private transient List<AtomicLong> taskLastRunDurationMs;
+
+ LockRemoverOperator(
+ StreamOperatorParameters<Void> parameters,
+ OperatorEventGateway operatorEventGateway,
+ String tableName,
+ List<String> maintenanceTaskNames) {
+ super(parameters);
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ }
+
+ @Override
+ public void open() throws Exception {
+ this.succeededTaskResultCounters =
+ Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.failedTaskResultCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.taskLastRunDurationMs = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ MetricGroup taskMetricGroup =
+ TableMaintenanceMetrics.groupFor(
+ getRuntimeContext(), tableName, maintenanceTaskNames.get(taskIndex), taskIndex);
+ succeededTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.SUCCEEDED_TASK_COUNTER));
+ failedTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.FAILED_TASK_COUNTER));
+ AtomicLong duration = new AtomicLong(0);
+ taskLastRunDurationMs.add(duration);
+ taskMetricGroup.gauge(TableMaintenanceMetrics.LAST_RUN_DURATION_MS, duration::get);
+ }
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ // no incoming events
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TaskResult> streamRecord) {
+ TaskResult taskResult = streamRecord.getValue();
+ LOG.info(
+ "Processing result {} for task {}",
+ taskResult,
+ maintenanceTaskNames.get(taskResult.taskIndex()));
+ long duration = System.currentTimeMillis() - taskResult.startEpoch();
+ operatorEventGateway.sendEventToCoordinator(
+ new LockReleasedEvent(tableName, streamRecord.getTimestamp()));
Review Comment:
One of the `LockReleasedEvent` is unnecessary
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverOperator.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.iceberg.flink.maintenance.api.TaskResult;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class LockRemoverOperator extends AbstractStreamOperator<Void>
+ implements OneInputStreamOperator<TaskResult, Void>, OperatorEventHandler {
+ private static final Logger LOG = LoggerFactory.getLogger(LockRemoverOperator.class);
+
+ private static final long serialVersionUID = 1L;
+
+ private final String tableName;
+ private final OperatorEventGateway operatorEventGateway;
+ private final List<String> maintenanceTaskNames;
+ private transient List<Counter> succeededTaskResultCounters;
+ private transient List<Counter> failedTaskResultCounters;
+ private transient List<AtomicLong> taskLastRunDurationMs;
+
+ LockRemoverOperator(
+ StreamOperatorParameters<Void> parameters,
+ OperatorEventGateway operatorEventGateway,
+ String tableName,
+ List<String> maintenanceTaskNames) {
+ super(parameters);
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ }
+
+ @Override
+ public void open() throws Exception {
+ this.succeededTaskResultCounters =
+ Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.failedTaskResultCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.taskLastRunDurationMs = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ MetricGroup taskMetricGroup =
+ TableMaintenanceMetrics.groupFor(
+ getRuntimeContext(), tableName, maintenanceTaskNames.get(taskIndex), taskIndex);
+ succeededTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.SUCCEEDED_TASK_COUNTER));
+ failedTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.FAILED_TASK_COUNTER));
+ AtomicLong duration = new AtomicLong(0);
+ taskLastRunDurationMs.add(duration);
+ taskMetricGroup.gauge(TableMaintenanceMetrics.LAST_RUN_DURATION_MS, duration::get);
+ }
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ // no incoming events
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TaskResult> streamRecord) {
+ TaskResult taskResult = streamRecord.getValue();
+ LOG.info(
+ "Processing result {} for task {}",
+ taskResult,
+ maintenanceTaskNames.get(taskResult.taskIndex()));
+ long duration = System.currentTimeMillis() - taskResult.startEpoch();
+ operatorEventGateway.sendEventToCoordinator(
+ new LockReleasedEvent(tableName, streamRecord.getTimestamp()));
+
+ // Update the metrics
+ taskLastRunDurationMs.get(taskResult.taskIndex()).set(duration);
+ if (taskResult.success()) {
+ succeededTaskResultCounters.get(taskResult.taskIndex()).inc();
+ } else {
+ failedTaskResultCounters.get(taskResult.taskIndex()).inc();
+ }
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ operatorEventGateway.sendEventToCoordinator(
Review Comment:
One of the `LockReleasedEvent` is unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2735788672
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ if (!pendingReleaseEvents.isEmpty()) {
+ pendingReleaseEvents.forEach(this::handleReleaseLock);
+ pendingReleaseEvents.clear();
+ }
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ @VisibleForTesting
+ void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ pendingReleaseEvents.add(lockReleasedEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
Review Comment:
Why don't we checkpoint all the locks? This will give us a precise snapshot of the current locking state. That way, we could also get rid of the recovery lock.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
Review Comment:
```suggestion
private boolean started;
```
Same here, `transient` isn't required.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
Review Comment:
I think we should use `final` here instead.
```suggestion
private final List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
```
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
Review Comment:
Have we considered a different flow, where we confirm the lock registration? Why are we assuming that every lock register event succeeds?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
+ * otherwise, it schedules the next attempt. When the job recovers from a failure, tasks from
+ * different execution paths of the previous run may still be running. Therefore, it first needs to
+ * send a lock with the maximum timestamp, and then send a recovery trigger. Only after the
+ * downstream removes this lock can we be sure that all tasks have fully stopped.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient boolean restoreTasks = false;
+ private transient boolean lockHeld = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
Review Comment:
AFAIK We register once on state restore, but never again, even after the lock has been released again. Am I missing something here?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
Review Comment:
There is classloader-level isolation between coordinators from different jobs, which means that there will be multiple instances of the static field because every job will load this class through its user class loader.
+1 for checking for existing consumers though. Is the locking meant to be re-entrant? Even so, we should check that its actually re-entrant and not coming from a different subtask.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
Review Comment:
I don't understand the method name. This method handles `lockRegisterEvent`. Would something like `registerLock(LockRegisterEvent event)` be better?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ if (!pendingReleaseEvents.isEmpty()) {
+ pendingReleaseEvents.forEach(this::handleReleaseLock);
+ pendingReleaseEvents.clear();
+ }
Review Comment:
AFAIK The check isn't required.
```suggestion
pendingReleaseEvents.forEach(this::handleReleaseLock);
pendingReleaseEvents.clear();
```
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
Review Comment:
```suggestion
private List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
```
`transient` doesn't make sense in this context. This class is not serializable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2746765011
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverOperatorFactory.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperatorFactory;
+import org.apache.flink.streaming.api.operators.CoordinatedOperatorFactory;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperatorFactory;
+import org.apache.flink.streaming.api.operators.StreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.iceberg.flink.maintenance.api.TaskResult;
+
+@Internal
+public class LockRemoverOperatorFactory extends AbstractStreamOperatorFactory<Void>
+ implements CoordinatedOperatorFactory<Void>, OneInputStreamOperatorFactory<TaskResult, Void> {
+ private final String tableName;
+ private final List<String> maintenanceTaskNames;
+
+ public LockRemoverOperatorFactory(String tableName, List<String> maintenanceTaskNames) {
+ this.tableName = tableName;
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ }
+
+ @Override
+ public OperatorCoordinator.Provider getCoordinatorProvider(
+ String operatorName, OperatorID operatorID) {
+ return new TableMaintenanceCoordinatorProvider(operatorName, operatorID);
Review Comment:
I noticed that we're now using the same coordinator code for both the TriggerManager and the LockRemover coordinator, albeit parameterized with the actual operator name. I found that confusing because the code executes different branches depending on which role it fulfills.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821293865
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
Review Comment:
Remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2832069912
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
Review Comment:
And this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2813038093
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
Review Comment:
Based on our discussion, we’ll move forward with this PR using synchronization locks for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2829265473
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ protected static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ protected static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, @Nullable byte[] checkpointData)
+ throws Exception {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
+
+ @SuppressWarnings("unchecked")
+ protected SubtaskGateways(String operatorName, int parallelism) {
+ this.operatorName = operatorName;
+ gateways = new Map[parallelism];
+
+ for (int i = 0; i < parallelism; ++i) {
+ gateways[i] = new java.util.HashMap<>();
+ }
+ }
+
+ protected void registerSubtaskGateway(SubtaskGateway gateway) {
+ int subtaskIndex = gateway.getSubtask();
+ int attemptNumber = gateway.getExecution().getAttemptNumber();
+ Preconditions.checkState(
+ !gateways[subtaskIndex].containsKey(attemptNumber),
+ "Coordinator of %s already has a subtask gateway for %d (#%d)",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ LOG.debug(
+ "Coordinator of {} registers gateway for subtask {} attempt {}",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ gateways[subtaskIndex].put(attemptNumber, gateway);
+ LOG.debug("Registered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected void unregisterSubtaskGateway(int subtaskIndex, int attemptNumber) {
+ gateways[subtaskIndex].remove(attemptNumber);
+ LOG.debug("Unregistered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected SubtaskGateway getSubtaskGateway(int subtaskIndex) {
+ Preconditions.checkState(
+ !gateways[subtaskIndex].isEmpty(),
+ "Coordinator subtask %d is not ready yet to receive events",
+ subtaskIndex);
+ return gateways[subtaskIndex].values().iterator().next();
+ }
+
+ protected void reset(int subtaskIndex) {
+ gateways[subtaskIndex].clear();
+ }
+ }
+
+ /** Custom thread factory for the coordinator executor. */
+ protected static class CoordinatorExecutorThreadFactory
+ implements java.util.concurrent.ThreadFactory, Thread.UncaughtExceptionHandler {
Review Comment:
It was added by IDEA during the migration; I’ve already removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818081791
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
Review Comment:
My understanding is that state restoration happens first, and only after the coordinator is started do we get any message exchanges, so this part should be OK.
In https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java, the constructor calls the `resetToCheckpoint`-related methods to restore state, and then `startScheduling()` calls `start` to start the coordinator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2815705969
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
Review Comment:
Can we make this a regular map as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816086186
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverCoordinator.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Coordinator for LockRemoverOperator. Handles lock release events from downstream operators. */
+@Experimental
+@Internal
+public class LockRemoverCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(LockRemoverCoordinator.class);
+
+ public LockRemoverCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created LockRemoverCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockReleaseEvent) {
+ handleReleaseLock((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
Review Comment:
Can we have this in the base?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2803050167
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
Review Comment:
That would work fine if we were using the same coordinator, but we have two coordinators. Unless we ensure that only one of the two coordinators processes events at the same time, the above assumption doesn't hold. Consider this scenario:
1. TriggerManagerOperator sends LockRegisterEvent
2. Maintenance tasks running
3. Maintenance task finished
4. LockRemoverOperator sends LockReleaseEvent
5. TriggerManagerCoordinator processes LockRegisterEvent
6. LockRemoverCoordinator processes LockReleaseEvent
This results in a deadlock because we only acquire the lock in TriggerManagerOperator during initializeState().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2846001427
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockRemoveCoordinator.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.concurrent.ExecutionException;
+import org.apache.flink.runtime.operators.coordination.EventReceivingTasks;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.api.parallel.ExecutionMode;
+
+@Timeout(value = 10)
+@Execution(ExecutionMode.SAME_THREAD)
+class TestLockRemoveCoordinator extends TestBaseCoordinator {
+
+ private EventReceivingTasks receivingTasks;
+
+ @BeforeEach
+ void before() {
+ this.receivingTasks = EventReceivingTasks.createForRunningTasks();
+ }
+
+ @Test
Review Comment:
not needed to be public
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#issuecomment-3952691612
Merged to main.
Well done @Guosmilesmile!
Thanks @mxm for the idea and the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818131269
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockTime = null;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleaseEvent event) {
+ if (lockTime == null) {
+ return;
+ }
+
+ if (event.timestamp() >= lockTime) {
Review Comment:
I think Flink’s mailbox threading model ensures these methods are executed serially on the same thread, so I didn’t add any synchronization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821339874
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockRemoverOperation.java:
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.FAILED_TASK_COUNTER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.LAST_RUN_DURATION_MS;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.SUCCEEDED_TASK_COUNTER;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.operators.coordination.MockOperatorEventGateway;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.flink.maintenance.api.TaskResult;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+@Timeout(value = 10)
+class TestLockRemoverOperation extends OperatorTestBase {
+ private static final String[] TASKS = new String[] {"task0", "task1", "task2"};
+ private static final String OPERATOR_NAME = "TestCoordinator";
+ private static final OperatorID TEST_OPERATOR_ID = new OperatorID(1234L, 5678L);
+
+ private LockRemoverCoordinator lockRemoverCoordinator;
+
+ @BeforeEach
+ public void before() {
+ MetricsReporterFactoryForTests.reset();
+ this.lockRemoverCoordinator = createCoordinator();
+ try {
+ lockRemoverCoordinator.start();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ super.after();
+ try {
+ lockRemoverCoordinator.close();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ void testProcess() throws Exception {
+ MockOperatorEventGateway mockGateway = new MockOperatorEventGateway();
+ LockRemoverOperator operator =
+ new LockRemoverOperator(null, mockGateway, DUMMY_TASK_NAME, Lists.newArrayList(TASKS[0]));
+ try (OneInputStreamOperatorTestHarness<TaskResult, Void> testHarness =
+ createHarness(operator)) {
+
+ testHarness.processElement(
+ new StreamRecord<>(new TaskResult(0, 0L, true, Lists.newArrayList())));
+ assertThat(mockGateway.getEventsSent()).hasSize(0);
+
+ testHarness.processWatermark(WATERMARK);
+ assertThat(mockGateway.getEventsSent()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TaskResult> source = new ManualSource<>(env, TypeInformation.of(TaskResult.class));
+ source
+ .dataStream()
+ .transform(
+ DUMMY_TASK_NAME,
+ TypeInformation.of(Void.class),
+ new LockRemoverOperatorFactory(DUMMY_TABLE_NAME, Lists.newArrayList(TASKS)))
+ .forceNonParallel();
+
+ JobClient jobClient = null;
+ long time = System.currentTimeMillis();
+ try {
+ jobClient = env.executeAsync();
+ // Start the 2 successful and one failed result trigger for task1, and 3 successful for task2
+ processAndCheck(source, new TaskResult(0, time, true, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(1, 0L, true, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(1, 0L, true, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(0, time, false, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(0, time, true, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(1, 0L, true, Lists.newArrayList()));
+
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(
+ DUMMY_TASK_NAME,
+ DUMMY_TABLE_NAME,
+ TASKS[1],
+ "1",
+ SUCCEEDED_TASK_COUNTER))
+ .equals(3L));
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[0], "0", SUCCEEDED_TASK_COUNTER),
+ 2L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[0], "0", FAILED_TASK_COUNTER),
+ 1L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[1], "1", SUCCEEDED_TASK_COUNTER),
+ 3L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[1], "1", FAILED_TASK_COUNTER),
+ 0L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[2], "2", SUCCEEDED_TASK_COUNTER),
+ 0L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[2], "2", FAILED_TASK_COUNTER),
+ 0L)
+ .build());
+
+ assertThat(
+ MetricsReporterFactoryForTests.gauge(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[0], "0", LAST_RUN_DURATION_MS)))
+ .isPositive();
+ assertThat(
+ MetricsReporterFactoryForTests.gauge(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[1], "1", LAST_RUN_DURATION_MS)))
+ .isGreaterThan(time);
+ assertThat(
+ MetricsReporterFactoryForTests.gauge(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[2], "2", LAST_RUN_DURATION_MS)))
+ .isZero();
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private void processAndCheck(ManualSource<TaskResult> source, TaskResult input) {
+ processAndCheck(source, input, null);
+ }
+
+ private void processAndCheck(
+ ManualSource<TaskResult> source, TaskResult input, String counterPrefix) {
Review Comment:
Do we need this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821344254
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManagerCoordinator.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.EventReceivingTasks;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.api.parallel.ExecutionMode;
+
+@Timeout(value = 10)
+@Execution(ExecutionMode.SAME_THREAD)
Review Comment:
Could you please explain this to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2836106889
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
Review Comment:
Since we move to base, we don't need it . Remove it .
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
Review Comment:
Since we move to base, we don't need it . Remove it .
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
Review Comment:
RIght
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2734901865
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
+ * otherwise, it schedules the next attempt. When the job recovers from a failure, tasks from
+ * different execution paths of the previous run may still be running. Therefore, it first needs to
+ * send a lock with the maximum timestamp, and then send a recovery trigger. Only after the
+ * downstream removes this lock can we be sure that all tasks have fully stopped.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient boolean restoreTasks = false;
+ private transient boolean lockHeld = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // - fire a recovery trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ restoreTasks = true;
+ lockHeld = false;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ restoreTasks = false;
+ lockHeld = false;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleasedEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ super.processWatermark(mark);
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockHeld = false;
+ this.restoreTasks = false;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleasedEvent event) {
+ if (event.lockId().equals(tableName)) {
Review Comment:
I found I have check it in coordinator, we don't need this check again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#issuecomment-3884695146
Hi @mxm , If you have time , help to review it again , thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2799683624
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
Review Comment:
Can the current `PENDING_RELEASE_EVENTS` handle out-of-order scenarios?
For example, suppose `TriggerManagerOperator` sends a registration message to `TriggerManagerCoordinator`, but the registration hasn’t completed yet. Then `LockRemoverOperator` sends a lock-release message to `LockRemoverCoordinator`. If it finds that the corresponding registration doesn’t exist yet, it will add the event to the static `PENDING_RELEASE_EVENTS`. After `TriggerManagerCoordinator` finishes the registration, it will iterate over the data in `PENDING_RELEASE_EVENTS` and trigger the lock release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2746293963
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ if (!pendingReleaseEvents.isEmpty()) {
+ pendingReleaseEvents.forEach(this::handleReleaseLock);
+ pendingReleaseEvents.clear();
+ }
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ @VisibleForTesting
+ void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ pendingReleaseEvents.add(lockReleasedEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
Review Comment:
Regarding recovery strategies for stateful tasks, these are two ideas:
1. Current approach
The `TriggerManagerOperator` does not persist the historical lock state. Instead, upon recovery it acquires a new lock and sends a recover trigger downstream. Once all tasks have finished, the downstream will release the lock.
2. Alternative approach
If we persist the lock state in `TriggerManagerOperator`, then upon task recovery there are two scenarios:
1. Stateless tasks will not release the lock.
2. Stateful tasks will release the lock when the task completes.
In that case, we would need to determine explicitly, based on the task type, whether we should manually trigger a lock release during recovery. Stateful tasks must not trigger this manually. This feels like it would couple the design more tightly to the task implementation, and each task would need to be modified. By contrast, approach 1 looks more generic and reusable.
Is my understanding correct? I’d like to hear your thoughts.@mxm @pvary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2746231445
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ if (!pendingReleaseEvents.isEmpty()) {
+ pendingReleaseEvents.forEach(this::handleReleaseLock);
+ pendingReleaseEvents.clear();
+ }
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ @VisibleForTesting
+ void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ pendingReleaseEvents.add(lockReleasedEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
Review Comment:
https://github.com/Guosmilesmile/iceberg/blob/14c82af75296a63b8552a6fc01f88c8e7713f627/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java#L176-L185
In this PR, we only have `lockTime` instead of a boolean flag. After restore, I reset the lock time to the current time in the `TriggerManagerOperator`, which is equivalent to releasing the lock. Therefore, there is no need to release the lock again from the coordinator; it has already been handled inside `TriggerManagerOperator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822381049
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ protected static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ protected static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
Review Comment:
We don’t need this—it's a leftover from before and has been removed.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManagerOperator to TriggerManagerCoordinator to register a lock release
+ * handler. This handler will be used to forward lock release events back to the operator when
+ * triggered by downstream operators.
+ */
+public class LockRegisterEvent implements OperatorEvent {
+
+ private final String lockId;
+
+ public LockRegisterEvent(String lockId) {
+ this.lockId = lockId;
+ }
+
+ public String lockId() {
+ return lockId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (!(o instanceof LockRegisterEvent)) {
+ return false;
+ }
+
+ LockRegisterEvent that = (LockRegisterEvent) o;
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816069219
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ protected static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ protected static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS =
+ new CopyOnWriteArrayList<>();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
Review Comment:
Originally, `handleReleaseLock` was in `LRCoordinator`, but because a pending release lock needs to be handled once after `TMCoordinator` registers, I moved this method into the base class.
For consistency and uniformity, I’d prefer to move both methods into the base class so the related logic is kept in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816025546
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -76,8 +76,12 @@ private void registerLock(LockRegisterEvent lockRegisterEvent) {
subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
});
- PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
- PENDING_RELEASE_EVENTS.clear();
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
Review Comment:
Good catch . Fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816129541
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverCoordinatorProvider.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator;
+
+@Internal
+public class LockRemoverCoordinatorProvider extends RecreateOnResetOperatorCoordinator.Provider {
Review Comment:
nit: Could this be just an internal class for `LockRemoverOperatorFactory`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816225559
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockTime = null;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleaseEvent event) {
Review Comment:
nit: handleLockRelease?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2836106541
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
Review Comment:
Chang it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2836110924
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
Review Comment:
Yes, we’ve restricted the parallelism to 1, so I removed the array here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818156748
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverCoordinatorProvider.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator;
+
+@Internal
+public class LockRemoverCoordinatorProvider extends RecreateOnResetOperatorCoordinator.Provider {
Review Comment:
Yes, move it .
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
Review Comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821365060
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockRemoveCoordinator.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import org.apache.flink.runtime.operators.coordination.EventReceivingTasks;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.api.parallel.ExecutionMode;
+
+@Timeout(value = 10)
+@Execution(ExecutionMode.SAME_THREAD)
+class TestLockRemoveCoordinator extends TestBaseCoordinator {
+
+ private EventReceivingTasks receivingTasks;
+
+ @BeforeEach
+ public void before() {
+ this.receivingTasks = EventReceivingTasks.createForRunningTasks();
+ }
+
+ @Test
+ public void testEventHandling() throws Exception {
+ try (LockRemoverCoordinator lockRemoverCoordinator =
+ new LockRemoverCoordinator(
+ OPERATOR_NAME, new MockOperatorCoordinatorContext(TEST_OPERATOR_ID, 1)); ) {
Review Comment:
```suggestion
OPERATOR_NAME, new MockOperatorCoordinatorContext(TEST_OPERATOR_ID, 1))) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2803454199
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
Review Comment:
Thanks for the explanation—I added an ACK mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2746146480
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
Review Comment:
You are right, I still keep the old state , I have change it to new version. Thanks for point it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2746757865
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ if (!pendingReleaseEvents.isEmpty()) {
+ pendingReleaseEvents.forEach(this::handleReleaseLock);
+ pendingReleaseEvents.clear();
+ }
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ @VisibleForTesting
+ void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ pendingReleaseEvents.add(lockReleasedEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
Review Comment:
I hadn't considered stateless tasks as much as stateful tasks, but I don't think it is a bad idea to make all tasks stateful in the sense that they are guaranteed to send a lock on restore. Maybe using a recovery approach is simpler after all, I just didn't find it to be very straight-forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2734903211
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
+ * otherwise, it schedules the next attempt. When the job recovers from a failure, tasks from
+ * different execution paths of the previous run may still be running. Therefore, it first needs to
+ * send a lock with the maximum timestamp, and then send a recovery trigger. Only after the
+ * downstream removes this lock can we be sure that all tasks have fully stopped.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient boolean restoreTasks = false;
+ private transient boolean lockHeld = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // - fire a recovery trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ restoreTasks = true;
+ lockHeld = false;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ restoreTasks = false;
+ lockHeld = false;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleasedEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ super.processWatermark(mark);
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockHeld = false;
+ this.restoreTasks = false;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleasedEvent event) {
+ if (event.lockId().equals(tableName)) {
+ this.lockHeld = false;
+ this.restoreTasks = false;
Review Comment:
I add a pendingReleaseEvents in this case , and when we registerTriggerManagerReceiveReleaseEvent will replay the pendingReleaseEvents again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2764303834
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
Review Comment:
I create a `BaseCoordinator` and keep the static map in `BaseCoordinator`, then have`TriggerManagerOperatorCoordinator` and `LockRemoverOperatorCoordinator` extend `BaseCoordinator` and implement their respective logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2804911717
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
Review Comment:
Could we just synchronize on the PENDING_RELEASE_EVENTS when putting something there, and when removing something there?
Ack seems like an overkill. But I might miss something, as I didn't have time to think through this from every aspects
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821265599
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManagerOperator to TriggerManagerCoordinator to register a lock release
+ * handler. This handler will be used to forward lock release events back to the operator when
+ * triggered by downstream operators.
+ */
+public class LockRegisterEvent implements OperatorEvent {
+
+ private final String lockId;
+
+ public LockRegisterEvent(String lockId) {
+ this.lockId = lockId;
+ }
+
+ public String lockId() {
+ return lockId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (!(o instanceof LockRegisterEvent)) {
+ return false;
+ }
+
+ LockRegisterEvent that = (LockRegisterEvent) o;
Review Comment:
```suggestion
if (!(o instanceof LockRegisterEvent that)) {
return false;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822407426
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestTableMaintenanceCoordinationLock.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.api;
+
+import static org.apache.iceberg.flink.SimpleDataUtil.createRowData;
+import static org.apache.iceberg.flink.maintenance.api.TableMaintenance.SOURCE_OPERATOR_NAME_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.Collections;
+import java.util.List;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.transformations.SourceTransformation;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.flink.maintenance.operator.ManualSource;
+import org.apache.iceberg.flink.maintenance.operator.OperatorTestBase;
+import org.apache.iceberg.flink.maintenance.operator.TableChange;
+import org.apache.iceberg.flink.sink.FlinkSink;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+class TestTableMaintenanceCoordinationLock extends OperatorTestBase {
+ private static final String MAINTENANCE_TASK_NAME = "TestTableMaintenance";
+ private static final String[] TASKS =
+ new String[] {MAINTENANCE_TASK_NAME + " [0]", MAINTENANCE_TASK_NAME + " [1]"};
+ private static final TableChange DUMMY_CHANGE = TableChange.builder().commitCount(1).build();
+ private static final List<Trigger> PROCESSED =
+ Collections.synchronizedList(Lists.newArrayListWithCapacity(1));
+
+ private StreamExecutionEnvironment env;
+ private Table table;
+
+ @TempDir private File checkpointDir;
+
+ @BeforeEach
+ public void beforeEach() throws IOException {
+ Configuration config = new Configuration();
+ config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
+ config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "file://" + checkpointDir.getPath());
+ this.env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ this.table = createTable();
+ insert(table, 1, "a");
+
+ PROCESSED.clear();
+ MaintenanceTaskBuilderForTest.counter = 0;
+ }
+
+ @Test
+ void testForChangeStream() throws Exception {
+ ManualSource<TableChange> schedulerSource =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+
+ TableMaintenance.Builder streamBuilder =
+ TableMaintenance.forChangeStream(schedulerSource.dataStream(), tableLoader())
+ .rateLimit(Duration.ofMillis(2))
+ .lockCheckDelay(Duration.ofSeconds(3))
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .scheduleOnDataFileCount(2)
+ .scheduleOnDataFileSize(3L)
+ .scheduleOnEqDeleteFileCount(4)
+ .scheduleOnEqDeleteRecordCount(5L)
+ .scheduleOnPosDeleteFileCount(6)
+ .scheduleOnPosDeleteRecordCount(7L)
+ .scheduleOnInterval(Duration.ofHours(1)));
+
+ sendEvents(schedulerSource, streamBuilder, ImmutableList.of(Tuple2.of(DUMMY_CHANGE, 1)));
+ }
+
+ @Test
+ void testForTable() throws Exception {
+ TableLoader tableLoader = tableLoader();
+
+ env.enableCheckpointing(10);
+
+ TableMaintenance.forTable(env, tableLoader)
+ .rateLimit(Duration.ofMillis(2))
+ .maxReadBack(2)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(2))
+ .append();
+
+ // Creating a stream for inserting data into the table concurrently
+ ManualSource<RowData> insertSource =
+ new ManualSource<>(env, InternalTypeInfo.of(FlinkSchemaUtil.convert(table.schema())));
+ FlinkSink.forRowData(insertSource.dataStream())
+ .tableLoader(tableLoader)
+ .uidPrefix(UID_SUFFIX + "-iceberg-sink")
+ .append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ insertSource.sendRecord(createRowData(2, "b"));
+
+ Awaitility.await().until(() -> PROCESSED.size() == 1);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testUidAndSlotSharingGroup() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupUnset() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, null);
+ checkSlotSharingGroupsAreSet(env, null);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupInherit() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupOverWrite() throws IOException {
+ String anotherUid = "Another-UID";
+ String anotherSlotSharingGroup = "Another-SlotSharingGroup";
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(anotherUid)
+ .slotSharingGroup(anotherSlotSharingGroup))
+ .append();
+
+ // Choose an operator from the scheduler part of the graph
+ Transformation<?> schedulerTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().equals("Trigger manager"))
+ .findFirst()
+ .orElseThrow();
+ assertThat(schedulerTransformation.getUid()).contains(UID_SUFFIX);
+ assertThat(schedulerTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(schedulerTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(SLOT_SHARING_GROUP);
+
+ // Choose an operator from the maintenance task part of the graph
+ Transformation<?> scheduledTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().startsWith(MAINTENANCE_TASK_NAME))
+ .findFirst()
+ .orElseThrow();
+ assertThat(scheduledTransformation.getUid()).contains(anotherUid);
+ assertThat(scheduledTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(scheduledTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(anotherSlotSharingGroup);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupForMonitorSource() throws IOException {
+ TableMaintenance.forTable(env, tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ Transformation<?> source = monitorSource();
+ assertThat(source).isNotNull();
+ assertThat(source.getUid()).contains(UID_SUFFIX);
+ assertThat(source.getSlotSharingGroup()).isPresent();
+ assertThat(source.getSlotSharingGroup().get().getName()).isEqualTo(SLOT_SHARING_GROUP);
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ /**
+ * Sends the events though the {@link ManualSource} provided, and waits until the given number of
+ * records are processed.
+ *
+ * @param schedulerSource used for sending the events
+ * @param streamBuilder used for generating the job
+ * @param eventsAndResultNumbers the pair of the event and the expected processed records
+ * @throws Exception if any
+ */
+ private void sendEvents(
Review Comment:
The same reason above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822409466
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManagerOperator.java:
##########
@@ -0,0 +1,679 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.operators.coordination.MockOperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+class TestTriggerManagerOperator extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String OPERATOR_NAME = "TestCoordinator";
+ private static final OperatorID TEST_OPERATOR_ID = new OperatorID(1234L, 5678L);
+ private static final String[] TASKS = new String[] {"task0", "task1"};
+ private long processingTime = 0L;
+ private String tableName;
+ private TriggerManagerCoordinator triggerManagerCoordinator;
+ private LockReleaseEvent lockReleaseEvent;
+
+ @BeforeEach
+ void before() {
+ super.before();
+ Table table = createTable();
+ this.tableName = table.name();
+ lockReleaseEvent = new LockReleaseEvent(tableName, 1L);
+ this.triggerManagerCoordinator = createCoordinator();
+ try {
+ triggerManagerCoordinator.start();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ super.after();
+ try {
+ triggerManagerCoordinator.close();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ void testCommitCount() throws Exception {
+ MockOperatorEventGateway mockGateway = new MockOperatorEventGateway();
+ TriggerManagerOperator operator =
+ createOperator(new TriggerEvaluator.Builder().commitCount(3).build(), mockGateway);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testDataFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().dataFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(1).build(), 0);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(5).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(2).build(), 4);
+ }
+ }
+
+ @Test
+ void testDataFileSizeInBytes() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().dataFileSizeInBytes(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testPosDeleteFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().posDeleteFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testPosDeleteRecordCount() throws Exception {
+
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().posDeleteRecordCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testEqDeleteFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().eqDeleteFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testEqDeleteRecordCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().eqDeleteRecordCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ TableChange event = TableChange.builder().dataFileCount(1).commitCount(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ operator.handleLockRelease(new LockReleaseEvent(tableName, newTime));
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ OperatorSubtaskState state;
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().commitCount(2).build(), new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.processElement(
+ TableChange.builder().dataFileCount(1).commitCount(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ TriggerManagerOperator newOperator =
+ createOperator(
+ new TriggerEvaluator.Builder().commitCount(2).build(), new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ new OneInputStreamOperatorTestHarness<>(newOperator)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Mock a recovery trigger lock
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ testHarness.processElement(TableChange.builder().commitCount(1).build(), EVENT_TIME_2);
+
+ // Remove the lock to allow the next trigger
+ newOperator.handleOperatorEvent(lockReleaseEvent);
+ testHarness.setProcessingTime(EVENT_TIME_2);
+
+ // At this point the output contains the recovery trigger and the real trigger
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ tableName,
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new MockOperatorEventGateway(),
+ DELAY,
+ 1);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLockCheckDelay() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ tableName,
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new MockOperatorEventGateway(),
+ 1,
+ DELAY);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.open();
+
+ // Create a lock to prevent execution, and check that there is no result
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1, false);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1, false);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ operator.handleOperatorEvent(lockReleaseEvent);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManagerOperatorFactory triggerManagerOperatorFactory =
+ new TriggerManagerOperatorFactory(
+ tableName,
+ Lists.newArrayList(TASKS),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new TriggerEvaluator.Builder().commitCount(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(
+ DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), triggerManagerOperatorFactory)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitCount(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER));
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitCount(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED)))
+ .isEqualTo(1L);
+
+ // manual unlock
+ triggerManagerCoordinator.handleReleaseLock(new LockReleaseEvent(tableName, Long.MAX_VALUE));
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // manual unlock
+ triggerManagerCoordinator.handleReleaseLock(new LockReleaseEvent(tableName, Long.MAX_VALUE));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED)))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[1], "1", TRIGGERED)))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED), -1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED), -1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED), 2L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[1], "1", TRIGGERED), 1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER), 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManagerOperatorFactory manager = manager(1_000_000L, 1L);
+
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), manager)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ triggerManagerCoordinator.handleReleaseLock(lockReleaseEvent);
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED))
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManagerOperatorFactory manager = manager(1L, 1_000_000L);
+
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), manager)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED))
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED),
+ rateLimiterTrigger)
+ .put(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED),
+ concurrentRunTrigger)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED), 1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER), 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
Review Comment:
Sorry,remove it.
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManagerOperator.java:
##########
@@ -0,0 +1,679 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.operators.coordination.MockOperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+class TestTriggerManagerOperator extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String OPERATOR_NAME = "TestCoordinator";
+ private static final OperatorID TEST_OPERATOR_ID = new OperatorID(1234L, 5678L);
+ private static final String[] TASKS = new String[] {"task0", "task1"};
+ private long processingTime = 0L;
+ private String tableName;
+ private TriggerManagerCoordinator triggerManagerCoordinator;
+ private LockReleaseEvent lockReleaseEvent;
+
+ @BeforeEach
+ void before() {
+ super.before();
+ Table table = createTable();
+ this.tableName = table.name();
+ lockReleaseEvent = new LockReleaseEvent(tableName, 1L);
+ this.triggerManagerCoordinator = createCoordinator();
+ try {
+ triggerManagerCoordinator.start();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ super.after();
+ try {
+ triggerManagerCoordinator.close();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ void testCommitCount() throws Exception {
+ MockOperatorEventGateway mockGateway = new MockOperatorEventGateway();
+ TriggerManagerOperator operator =
+ createOperator(new TriggerEvaluator.Builder().commitCount(3).build(), mockGateway);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testDataFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().dataFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(1).build(), 0);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(5).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(2).build(), 4);
+ }
+ }
+
+ @Test
+ void testDataFileSizeInBytes() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().dataFileSizeInBytes(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testPosDeleteFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().posDeleteFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testPosDeleteRecordCount() throws Exception {
+
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().posDeleteRecordCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testEqDeleteFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().eqDeleteFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testEqDeleteRecordCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().eqDeleteRecordCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ TableChange event = TableChange.builder().dataFileCount(1).commitCount(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ operator.handleLockRelease(new LockReleaseEvent(tableName, newTime));
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ OperatorSubtaskState state;
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().commitCount(2).build(), new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.processElement(
+ TableChange.builder().dataFileCount(1).commitCount(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ TriggerManagerOperator newOperator =
+ createOperator(
+ new TriggerEvaluator.Builder().commitCount(2).build(), new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ new OneInputStreamOperatorTestHarness<>(newOperator)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Mock a recovery trigger lock
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ testHarness.processElement(TableChange.builder().commitCount(1).build(), EVENT_TIME_2);
+
+ // Remove the lock to allow the next trigger
+ newOperator.handleOperatorEvent(lockReleaseEvent);
+ testHarness.setProcessingTime(EVENT_TIME_2);
+
+ // At this point the output contains the recovery trigger and the real trigger
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ tableName,
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new MockOperatorEventGateway(),
+ DELAY,
+ 1);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLockCheckDelay() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ tableName,
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new MockOperatorEventGateway(),
+ 1,
+ DELAY);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.open();
+
+ // Create a lock to prevent execution, and check that there is no result
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1, false);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1, false);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ operator.handleOperatorEvent(lockReleaseEvent);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManagerOperatorFactory triggerManagerOperatorFactory =
+ new TriggerManagerOperatorFactory(
+ tableName,
+ Lists.newArrayList(TASKS),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new TriggerEvaluator.Builder().commitCount(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(
+ DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), triggerManagerOperatorFactory)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitCount(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER));
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitCount(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED)))
+ .isEqualTo(1L);
+
+ // manual unlock
+ triggerManagerCoordinator.handleReleaseLock(new LockReleaseEvent(tableName, Long.MAX_VALUE));
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // manual unlock
+ triggerManagerCoordinator.handleReleaseLock(new LockReleaseEvent(tableName, Long.MAX_VALUE));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED)))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[1], "1", TRIGGERED)))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED), -1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED), -1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED), 2L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[1], "1", TRIGGERED), 1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER), 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManagerOperatorFactory manager = manager(1_000_000L, 1L);
+
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), manager)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ triggerManagerCoordinator.handleReleaseLock(lockReleaseEvent);
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED))
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManagerOperatorFactory manager = manager(1L, 1_000_000L);
+
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), manager)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED))
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED),
+ rateLimiterTrigger)
+ .put(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED),
+ concurrentRunTrigger)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED), 1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER), 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ TriggerManagerOperator operator,
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ addEventAndCheckResult(operator, testHarness, event, expectedSize, true);
+ }
+
+ private void addEventAndCheckResult(
+ TriggerManagerOperator operator,
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize,
+ boolean removeLock)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ if (removeLock && operator.lockTime() != null) {
+ // Remove the lock to allow the next trigger
+ operator.handleLockRelease(new LockReleaseEvent(tableName, processingTime));
+ }
+ }
+
+ private TriggerManagerOperatorFactory manager(long minFireDelayMs, long lockCheckDelayMs) {
+ return new TriggerManagerOperatorFactory(
+ tableName,
+ Lists.newArrayList(TASKS[0]),
+ Lists.newArrayList(new TriggerEvaluator.Builder().commitCount(2).build()),
+ minFireDelayMs,
+ lockCheckDelayMs);
+ }
+
+ private static void assertTriggers(List<Trigger> expected, List<Trigger> actual) {
+ assertThat(actual).hasSize(expected.size());
+ for (int i = 0; i < expected.size(); ++i) {
+ Trigger expectedTrigger = expected.get(i);
+ Trigger actualTrigger = actual.get(i);
+ assertThat(actualTrigger.timestamp()).isEqualTo(expectedTrigger.timestamp());
+ assertThat(actualTrigger.taskId()).isEqualTo(expectedTrigger.taskId());
+ assertThat(actualTrigger.isRecovery()).isEqualTo(expectedTrigger.isRecovery());
+ }
+ }
+
+ private static TriggerManagerCoordinator createCoordinator() {
+ return new TriggerManagerCoordinator(
+ OPERATOR_NAME, new MockOperatorCoordinatorContext(TEST_OPERATOR_ID, 2));
+ }
+
+ private TriggerManagerOperator createOperator(
+ TriggerEvaluator evaluator, OperatorEventGateway mockGateway) {
+ return createOperator(tableName, evaluator, mockGateway, 1, 1);
+ }
+
+ private TriggerManagerOperator createOperator(
+ String lockId,
+ TriggerEvaluator evaluator,
+ OperatorEventGateway mockGateway,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ TriggerManagerOperator operator =
+ new TriggerManagerOperator(
+ null,
+ mockGateway,
+ Lists.newArrayList(TASKS[0]),
+ Lists.newArrayList(evaluator),
+ minFireDelayMs,
+ lockCheckDelayMs,
+ lockId);
+ return operator;
Review Comment:
Change it .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822408314
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockRemoverOperation.java:
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.FAILED_TASK_COUNTER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.LAST_RUN_DURATION_MS;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.SUCCEEDED_TASK_COUNTER;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.operators.coordination.MockOperatorEventGateway;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.flink.maintenance.api.TaskResult;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+@Timeout(value = 10)
+class TestLockRemoverOperation extends OperatorTestBase {
+ private static final String[] TASKS = new String[] {"task0", "task1", "task2"};
+ private static final String OPERATOR_NAME = "TestCoordinator";
+ private static final OperatorID TEST_OPERATOR_ID = new OperatorID(1234L, 5678L);
+
+ private LockRemoverCoordinator lockRemoverCoordinator;
+
+ @BeforeEach
+ public void before() {
+ MetricsReporterFactoryForTests.reset();
+ this.lockRemoverCoordinator = createCoordinator();
+ try {
+ lockRemoverCoordinator.start();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ super.after();
+ try {
+ lockRemoverCoordinator.close();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ void testProcess() throws Exception {
+ MockOperatorEventGateway mockGateway = new MockOperatorEventGateway();
+ LockRemoverOperator operator =
+ new LockRemoverOperator(null, mockGateway, DUMMY_TASK_NAME, Lists.newArrayList(TASKS[0]));
+ try (OneInputStreamOperatorTestHarness<TaskResult, Void> testHarness =
+ createHarness(operator)) {
+
+ testHarness.processElement(
+ new StreamRecord<>(new TaskResult(0, 0L, true, Lists.newArrayList())));
+ assertThat(mockGateway.getEventsSent()).hasSize(0);
+
+ testHarness.processWatermark(WATERMARK);
+ assertThat(mockGateway.getEventsSent()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TaskResult> source = new ManualSource<>(env, TypeInformation.of(TaskResult.class));
+ source
+ .dataStream()
+ .transform(
+ DUMMY_TASK_NAME,
+ TypeInformation.of(Void.class),
+ new LockRemoverOperatorFactory(DUMMY_TABLE_NAME, Lists.newArrayList(TASKS)))
+ .forceNonParallel();
+
+ JobClient jobClient = null;
+ long time = System.currentTimeMillis();
+ try {
+ jobClient = env.executeAsync();
+ // Start the 2 successful and one failed result trigger for task1, and 3 successful for task2
+ processAndCheck(source, new TaskResult(0, time, true, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(1, 0L, true, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(1, 0L, true, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(0, time, false, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(0, time, true, Lists.newArrayList()));
+ processAndCheck(source, new TaskResult(1, 0L, true, Lists.newArrayList()));
+
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(
+ DUMMY_TASK_NAME,
+ DUMMY_TABLE_NAME,
+ TASKS[1],
+ "1",
+ SUCCEEDED_TASK_COUNTER))
+ .equals(3L));
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[0], "0", SUCCEEDED_TASK_COUNTER),
+ 2L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[0], "0", FAILED_TASK_COUNTER),
+ 1L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[1], "1", SUCCEEDED_TASK_COUNTER),
+ 3L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[1], "1", FAILED_TASK_COUNTER),
+ 0L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[2], "2", SUCCEEDED_TASK_COUNTER),
+ 0L)
+ .put(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[2], "2", FAILED_TASK_COUNTER),
+ 0L)
+ .build());
+
+ assertThat(
+ MetricsReporterFactoryForTests.gauge(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[0], "0", LAST_RUN_DURATION_MS)))
+ .isPositive();
+ assertThat(
+ MetricsReporterFactoryForTests.gauge(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[1], "1", LAST_RUN_DURATION_MS)))
+ .isGreaterThan(time);
+ assertThat(
+ MetricsReporterFactoryForTests.gauge(
+ ImmutableList.of(
+ DUMMY_TASK_NAME, DUMMY_TABLE_NAME, TASKS[2], "2", LAST_RUN_DURATION_MS)))
+ .isZero();
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private void processAndCheck(ManualSource<TaskResult> source, TaskResult input) {
+ processAndCheck(source, input, null);
+ }
+
+ private void processAndCheck(
+ ManualSource<TaskResult> source, TaskResult input, String counterPrefix) {
Review Comment:
Combine it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818137168
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManager operator to TriggerManagerCoordinator to confirm that the lock has
+ * been acquired and the maintenance task is starting.
Review Comment:
Changed it now.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockReleaseEvent.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * Event sent from TriggerManagerOperatorCoordinator to TriggerManager operator to notify that a
+ * lock has been released (watermark arrived).
Review Comment:
Yes,change it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818135394
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManager operator to TriggerManagerCoordinator to confirm that the lock has
+ * been acquired and the maintenance task is starting.
+ */
+public class LockRegisterEvent implements OperatorEvent {
+
+ private final String lockId;
+ private final long timestamp;
+
+ public LockRegisterEvent(String lockId, long timestamp) {
+ this.lockId = lockId;
+ this.timestamp = timestamp;
+ }
+
+ public String lockId() {
+ return lockId;
+ }
+
+ public long timestamp() {
+ return timestamp;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (!(o instanceof LockRegisterEvent)) {
+ return false;
+ }
+
+ LockRegisterEvent that = (LockRegisterEvent) o;
+ return Objects.equal(timestamp, that.timestamp) && Objects.equal(lockId, that.lockId);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(timestamp, lockId);
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("timeStamp", timestamp)
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816246882
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockTime = null;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleaseEvent event) {
+ if (lockTime == null) {
+ return;
+ }
+
+ if (event.timestamp() >= lockTime) {
Review Comment:
Do we need to synchronize, or it is handled by Flink? The following could conflict:
- handleOperatorEvent - release lock
- initializeState - setting restore lock
- onProcessingTime - checking and setting lock
- processElement - checking and setting lock
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2836106371
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
+
+ @SuppressWarnings("unchecked")
+ protected SubtaskGateways(String operatorName, int parallelism) {
+ this.operatorName = operatorName;
+ gateways = new Map[parallelism];
+
+ for (int i = 0; i < parallelism; ++i) {
+ gateways[i] = new java.util.HashMap<>();
+ }
+ }
+
+ protected void registerSubtaskGateway(SubtaskGateway gateway) {
+ int subtaskIndex = gateway.getSubtask();
+ int attemptNumber = gateway.getExecution().getAttemptNumber();
+ Preconditions.checkState(
+ !gateways[subtaskIndex].containsKey(attemptNumber),
+ "Coordinator of %s already has a subtask gateway for %d (#%d)",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ LOG.debug(
+ "Coordinator of {} registers gateway for subtask {} attempt {}",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ gateways[subtaskIndex].put(attemptNumber, gateway);
+ LOG.debug("Registered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected void unregisterSubtaskGateway(int subtaskIndex, int attemptNumber) {
+ gateways[subtaskIndex].remove(attemptNumber);
+ LOG.debug("Unregistered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected SubtaskGateway getSubtaskGateway(int subtaskIndex) {
+ Preconditions.checkState(
+ !gateways[subtaskIndex].isEmpty(),
+ "Coordinator subtask %d is not ready yet to receive events",
+ subtaskIndex);
+ return gateways[subtaskIndex].values().iterator().next();
Review Comment:
This part is my mistake. We should align with Flink’s `SourceCoordinator` and use `getOnlyElement`, consistent with the implementation here:
https://github.com/apache/flink/blob/fdc2686e33d8dd1a338e3cc1432a6b8b2cad6d71/flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorContext.java#L725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2836106371
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
+
+ @SuppressWarnings("unchecked")
+ protected SubtaskGateways(String operatorName, int parallelism) {
+ this.operatorName = operatorName;
+ gateways = new Map[parallelism];
+
+ for (int i = 0; i < parallelism; ++i) {
+ gateways[i] = new java.util.HashMap<>();
+ }
+ }
+
+ protected void registerSubtaskGateway(SubtaskGateway gateway) {
+ int subtaskIndex = gateway.getSubtask();
+ int attemptNumber = gateway.getExecution().getAttemptNumber();
+ Preconditions.checkState(
+ !gateways[subtaskIndex].containsKey(attemptNumber),
+ "Coordinator of %s already has a subtask gateway for %d (#%d)",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ LOG.debug(
+ "Coordinator of {} registers gateway for subtask {} attempt {}",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ gateways[subtaskIndex].put(attemptNumber, gateway);
+ LOG.debug("Registered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected void unregisterSubtaskGateway(int subtaskIndex, int attemptNumber) {
+ gateways[subtaskIndex].remove(attemptNumber);
+ LOG.debug("Unregistered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected SubtaskGateway getSubtaskGateway(int subtaskIndex) {
+ Preconditions.checkState(
+ !gateways[subtaskIndex].isEmpty(),
+ "Coordinator subtask %d is not ready yet to receive events",
+ subtaskIndex);
+ return gateways[subtaskIndex].values().iterator().next();
Review Comment:
Good Catch. This part is my mistake. We should align with Flink’s `SourceCoordinator` and use `getOnlyElement`, consistent with the implementation here:
https://github.com/apache/flink/blob/fdc2686e33d8dd1a338e3cc1432a6b8b2cad6d71/flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorContext.java#L725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822405783
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestTableMaintenanceCoordinationLock.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.api;
+
+import static org.apache.iceberg.flink.SimpleDataUtil.createRowData;
+import static org.apache.iceberg.flink.maintenance.api.TableMaintenance.SOURCE_OPERATOR_NAME_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.Collections;
+import java.util.List;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.transformations.SourceTransformation;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.flink.maintenance.operator.ManualSource;
+import org.apache.iceberg.flink.maintenance.operator.OperatorTestBase;
+import org.apache.iceberg.flink.maintenance.operator.TableChange;
+import org.apache.iceberg.flink.sink.FlinkSink;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+class TestTableMaintenanceCoordinationLock extends OperatorTestBase {
+ private static final String MAINTENANCE_TASK_NAME = "TestTableMaintenance";
+ private static final String[] TASKS =
+ new String[] {MAINTENANCE_TASK_NAME + " [0]", MAINTENANCE_TASK_NAME + " [1]"};
+ private static final TableChange DUMMY_CHANGE = TableChange.builder().commitCount(1).build();
+ private static final List<Trigger> PROCESSED =
+ Collections.synchronizedList(Lists.newArrayListWithCapacity(1));
+
+ private StreamExecutionEnvironment env;
+ private Table table;
+
+ @TempDir private File checkpointDir;
+
+ @BeforeEach
+ public void beforeEach() throws IOException {
+ Configuration config = new Configuration();
+ config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
+ config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "file://" + checkpointDir.getPath());
+ this.env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ this.table = createTable();
+ insert(table, 1, "a");
+
+ PROCESSED.clear();
+ MaintenanceTaskBuilderForTest.counter = 0;
+ }
+
+ @Test
+ void testForChangeStream() throws Exception {
+ ManualSource<TableChange> schedulerSource =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+
+ TableMaintenance.Builder streamBuilder =
+ TableMaintenance.forChangeStream(schedulerSource.dataStream(), tableLoader())
+ .rateLimit(Duration.ofMillis(2))
+ .lockCheckDelay(Duration.ofSeconds(3))
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .scheduleOnDataFileCount(2)
+ .scheduleOnDataFileSize(3L)
+ .scheduleOnEqDeleteFileCount(4)
+ .scheduleOnEqDeleteRecordCount(5L)
+ .scheduleOnPosDeleteFileCount(6)
+ .scheduleOnPosDeleteRecordCount(7L)
+ .scheduleOnInterval(Duration.ofHours(1)));
+
+ sendEvents(schedulerSource, streamBuilder, ImmutableList.of(Tuple2.of(DUMMY_CHANGE, 1)));
+ }
+
+ @Test
+ void testForTable() throws Exception {
+ TableLoader tableLoader = tableLoader();
+
+ env.enableCheckpointing(10);
+
+ TableMaintenance.forTable(env, tableLoader)
+ .rateLimit(Duration.ofMillis(2))
+ .maxReadBack(2)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(2))
+ .append();
+
+ // Creating a stream for inserting data into the table concurrently
+ ManualSource<RowData> insertSource =
+ new ManualSource<>(env, InternalTypeInfo.of(FlinkSchemaUtil.convert(table.schema())));
+ FlinkSink.forRowData(insertSource.dataStream())
+ .tableLoader(tableLoader)
+ .uidPrefix(UID_SUFFIX + "-iceberg-sink")
+ .append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ insertSource.sendRecord(createRowData(2, "b"));
+
+ Awaitility.await().until(() -> PROCESSED.size() == 1);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testUidAndSlotSharingGroup() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupUnset() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, null);
+ checkSlotSharingGroupsAreSet(env, null);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupInherit() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupOverWrite() throws IOException {
+ String anotherUid = "Another-UID";
+ String anotherSlotSharingGroup = "Another-SlotSharingGroup";
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(anotherUid)
+ .slotSharingGroup(anotherSlotSharingGroup))
+ .append();
+
+ // Choose an operator from the scheduler part of the graph
+ Transformation<?> schedulerTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().equals("Trigger manager"))
+ .findFirst()
+ .orElseThrow();
+ assertThat(schedulerTransformation.getUid()).contains(UID_SUFFIX);
+ assertThat(schedulerTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(schedulerTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(SLOT_SHARING_GROUP);
+
+ // Choose an operator from the maintenance task part of the graph
+ Transformation<?> scheduledTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().startsWith(MAINTENANCE_TASK_NAME))
+ .findFirst()
+ .orElseThrow();
+ assertThat(scheduledTransformation.getUid()).contains(anotherUid);
+ assertThat(scheduledTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(scheduledTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(anotherSlotSharingGroup);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupForMonitorSource() throws IOException {
+ TableMaintenance.forTable(env, tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ Transformation<?> source = monitorSource();
+ assertThat(source).isNotNull();
+ assertThat(source.getUid()).contains(UID_SUFFIX);
+ assertThat(source.getSlotSharingGroup()).isPresent();
+ assertThat(source.getSlotSharingGroup().get().getName()).isEqualTo(SLOT_SHARING_GROUP);
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ /**
+ * Sends the events though the {@link ManualSource} provided, and waits until the given number of
+ * records are processed.
+ *
+ * @param schedulerSource used for sending the events
+ * @param streamBuilder used for generating the job
+ * @param eventsAndResultNumbers the pair of the event and the expected processed records
+ * @throws Exception if any
+ */
+ private void sendEvents(
+ ManualSource<TableChange> schedulerSource,
+ TableMaintenance.Builder streamBuilder,
+ List<Tuple2<TableChange, Integer>> eventsAndResultNumbers)
+ throws Exception {
+ streamBuilder.append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ eventsAndResultNumbers.forEach(
+ eventsAndResultNumber -> {
+ int expectedSize = PROCESSED.size() + eventsAndResultNumber.f1;
+ schedulerSource.sendRecord(eventsAndResultNumber.f0);
+ Awaitility.await().until(() -> PROCESSED.size() == expectedSize);
+ });
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ /**
+ * Finds the {@link org.apache.iceberg.flink.maintenance.operator.MonitorSource} for testing
+ * purposes by parsing the transformation tree.
+ *
+ * @return The monitor source if we found it
+ */
+ private Transformation<?> monitorSource() {
Review Comment:
Some of the code here is the same as in `TestTableMaintenance`, and we could extract the common parts into `MaintenanceTaskInfraExtension`. However, if we deprecate the in-memory lock later, then `TestTableMaintenance` will also be deprecated. Because of that, I didn’t do this refactor and instead made `TestTableMaintenanceCoordinationLock` independent.
If needed, we can refactor it later; for now I’m keeping it as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822378313
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ protected static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ protected static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
Review Comment:
Fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2832094320
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
+
+ @SuppressWarnings("unchecked")
+ protected SubtaskGateways(String operatorName, int parallelism) {
+ this.operatorName = operatorName;
+ gateways = new Map[parallelism];
+
+ for (int i = 0; i < parallelism; ++i) {
+ gateways[i] = new java.util.HashMap<>();
+ }
+ }
+
+ protected void registerSubtaskGateway(SubtaskGateway gateway) {
+ int subtaskIndex = gateway.getSubtask();
+ int attemptNumber = gateway.getExecution().getAttemptNumber();
+ Preconditions.checkState(
+ !gateways[subtaskIndex].containsKey(attemptNumber),
+ "Coordinator of %s already has a subtask gateway for %d (#%d)",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ LOG.debug(
+ "Coordinator of {} registers gateway for subtask {} attempt {}",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ gateways[subtaskIndex].put(attemptNumber, gateway);
+ LOG.debug("Registered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected void unregisterSubtaskGateway(int subtaskIndex, int attemptNumber) {
+ gateways[subtaskIndex].remove(attemptNumber);
+ LOG.debug("Unregistered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected SubtaskGateway getSubtaskGateway(int subtaskIndex) {
+ Preconditions.checkState(
+ !gateways[subtaskIndex].isEmpty(),
+ "Coordinator subtask %d is not ready yet to receive events",
+ subtaskIndex);
+ return gateways[subtaskIndex].values().iterator().next();
+ }
+
+ protected void reset(int subtaskIndex) {
+ gateways[subtaskIndex].clear();
+ }
+ }
+
+ /** Custom thread factory for the coordinator executor. */
+ protected static class CoordinatorExecutorThreadFactory
Review Comment:
Private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2815681390
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -76,8 +76,12 @@ private void registerLock(LockRegisterEvent lockRegisterEvent) {
subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
});
- PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
- PENDING_RELEASE_EVENTS.clear();
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
Review Comment:
I think this is not correct. You need to move the check inside the synchronized block.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
Review Comment:
This can be a normal `ArrayList` now.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
Review Comment:
Can we make this a regular map as well?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
Review Comment:
Why not make the whole method `synchronized`? I think this makes things a bit easier.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -81,11 +81,21 @@ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
lockReleaseEvent.lockId(),
lockReleaseEvent.timestamp());
} else {
Review Comment:
This block now gets repeated in the else `block` below. IMHO, we should just make the whole method synchronized to avoid those premature optimizations which make the code hard to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821329454
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestTableMaintenanceCoordinationLock.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.api;
+
+import static org.apache.iceberg.flink.SimpleDataUtil.createRowData;
+import static org.apache.iceberg.flink.maintenance.api.TableMaintenance.SOURCE_OPERATOR_NAME_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.Collections;
+import java.util.List;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.transformations.SourceTransformation;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.flink.maintenance.operator.ManualSource;
+import org.apache.iceberg.flink.maintenance.operator.OperatorTestBase;
+import org.apache.iceberg.flink.maintenance.operator.TableChange;
+import org.apache.iceberg.flink.sink.FlinkSink;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+class TestTableMaintenanceCoordinationLock extends OperatorTestBase {
+ private static final String MAINTENANCE_TASK_NAME = "TestTableMaintenance";
+ private static final String[] TASKS =
+ new String[] {MAINTENANCE_TASK_NAME + " [0]", MAINTENANCE_TASK_NAME + " [1]"};
+ private static final TableChange DUMMY_CHANGE = TableChange.builder().commitCount(1).build();
+ private static final List<Trigger> PROCESSED =
+ Collections.synchronizedList(Lists.newArrayListWithCapacity(1));
+
+ private StreamExecutionEnvironment env;
+ private Table table;
+
+ @TempDir private File checkpointDir;
+
+ @BeforeEach
+ public void beforeEach() throws IOException {
+ Configuration config = new Configuration();
+ config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
+ config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "file://" + checkpointDir.getPath());
+ this.env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ this.table = createTable();
+ insert(table, 1, "a");
+
+ PROCESSED.clear();
+ MaintenanceTaskBuilderForTest.counter = 0;
+ }
+
+ @Test
+ void testForChangeStream() throws Exception {
+ ManualSource<TableChange> schedulerSource =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+
+ TableMaintenance.Builder streamBuilder =
+ TableMaintenance.forChangeStream(schedulerSource.dataStream(), tableLoader())
+ .rateLimit(Duration.ofMillis(2))
+ .lockCheckDelay(Duration.ofSeconds(3))
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .scheduleOnDataFileCount(2)
+ .scheduleOnDataFileSize(3L)
+ .scheduleOnEqDeleteFileCount(4)
+ .scheduleOnEqDeleteRecordCount(5L)
+ .scheduleOnPosDeleteFileCount(6)
+ .scheduleOnPosDeleteRecordCount(7L)
+ .scheduleOnInterval(Duration.ofHours(1)));
+
+ sendEvents(schedulerSource, streamBuilder, ImmutableList.of(Tuple2.of(DUMMY_CHANGE, 1)));
+ }
+
+ @Test
+ void testForTable() throws Exception {
+ TableLoader tableLoader = tableLoader();
+
+ env.enableCheckpointing(10);
+
+ TableMaintenance.forTable(env, tableLoader)
+ .rateLimit(Duration.ofMillis(2))
+ .maxReadBack(2)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(2))
+ .append();
+
+ // Creating a stream for inserting data into the table concurrently
+ ManualSource<RowData> insertSource =
+ new ManualSource<>(env, InternalTypeInfo.of(FlinkSchemaUtil.convert(table.schema())));
+ FlinkSink.forRowData(insertSource.dataStream())
+ .tableLoader(tableLoader)
+ .uidPrefix(UID_SUFFIX + "-iceberg-sink")
+ .append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ insertSource.sendRecord(createRowData(2, "b"));
+
+ Awaitility.await().until(() -> PROCESSED.size() == 1);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testUidAndSlotSharingGroup() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupUnset() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, null);
+ checkSlotSharingGroupsAreSet(env, null);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupInherit() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupOverWrite() throws IOException {
+ String anotherUid = "Another-UID";
+ String anotherSlotSharingGroup = "Another-SlotSharingGroup";
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(anotherUid)
+ .slotSharingGroup(anotherSlotSharingGroup))
+ .append();
+
+ // Choose an operator from the scheduler part of the graph
+ Transformation<?> schedulerTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().equals("Trigger manager"))
+ .findFirst()
+ .orElseThrow();
+ assertThat(schedulerTransformation.getUid()).contains(UID_SUFFIX);
+ assertThat(schedulerTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(schedulerTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(SLOT_SHARING_GROUP);
+
+ // Choose an operator from the maintenance task part of the graph
+ Transformation<?> scheduledTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().startsWith(MAINTENANCE_TASK_NAME))
+ .findFirst()
+ .orElseThrow();
+ assertThat(scheduledTransformation.getUid()).contains(anotherUid);
+ assertThat(scheduledTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(scheduledTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(anotherSlotSharingGroup);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupForMonitorSource() throws IOException {
+ TableMaintenance.forTable(env, tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ Transformation<?> source = monitorSource();
+ assertThat(source).isNotNull();
+ assertThat(source.getUid()).contains(UID_SUFFIX);
+ assertThat(source.getSlotSharingGroup()).isPresent();
+ assertThat(source.getSlotSharingGroup().get().getName()).isEqualTo(SLOT_SHARING_GROUP);
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ /**
+ * Sends the events though the {@link ManualSource} provided, and waits until the given number of
+ * records are processed.
+ *
+ * @param schedulerSource used for sending the events
+ * @param streamBuilder used for generating the job
+ * @param eventsAndResultNumbers the pair of the event and the expected processed records
+ * @throws Exception if any
+ */
+ private void sendEvents(
+ ManualSource<TableChange> schedulerSource,
+ TableMaintenance.Builder streamBuilder,
+ List<Tuple2<TableChange, Integer>> eventsAndResultNumbers)
+ throws Exception {
+ streamBuilder.append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ eventsAndResultNumbers.forEach(
+ eventsAndResultNumber -> {
+ int expectedSize = PROCESSED.size() + eventsAndResultNumber.f1;
+ schedulerSource.sendRecord(eventsAndResultNumber.f0);
+ Awaitility.await().until(() -> PROCESSED.size() == expectedSize);
+ });
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ /**
+ * Finds the {@link org.apache.iceberg.flink.maintenance.operator.MonitorSource} for testing
+ * purposes by parsing the transformation tree.
+ *
+ * @return The monitor source if we found it
+ */
+ private Transformation<?> monitorSource() {
Review Comment:
Is it worth to use/refactor MaintenanceTaskInfraExtension and resue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821355115
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManagerOperator.java:
##########
@@ -0,0 +1,679 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.operators.coordination.MockOperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+class TestTriggerManagerOperator extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String OPERATOR_NAME = "TestCoordinator";
+ private static final OperatorID TEST_OPERATOR_ID = new OperatorID(1234L, 5678L);
+ private static final String[] TASKS = new String[] {"task0", "task1"};
+ private long processingTime = 0L;
+ private String tableName;
+ private TriggerManagerCoordinator triggerManagerCoordinator;
+ private LockReleaseEvent lockReleaseEvent;
+
+ @BeforeEach
+ void before() {
+ super.before();
+ Table table = createTable();
+ this.tableName = table.name();
+ lockReleaseEvent = new LockReleaseEvent(tableName, 1L);
+ this.triggerManagerCoordinator = createCoordinator();
+ try {
+ triggerManagerCoordinator.start();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ super.after();
+ try {
+ triggerManagerCoordinator.close();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ void testCommitCount() throws Exception {
+ MockOperatorEventGateway mockGateway = new MockOperatorEventGateway();
+ TriggerManagerOperator operator =
+ createOperator(new TriggerEvaluator.Builder().commitCount(3).build(), mockGateway);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testDataFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().dataFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(1).build(), 0);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(5).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(2).build(), 4);
+ }
+ }
+
+ @Test
+ void testDataFileSizeInBytes() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().dataFileSizeInBytes(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testPosDeleteFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().posDeleteFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testPosDeleteRecordCount() throws Exception {
+
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().posDeleteRecordCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testEqDeleteFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().eqDeleteFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testEqDeleteRecordCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().eqDeleteRecordCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ TableChange event = TableChange.builder().dataFileCount(1).commitCount(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ operator.handleLockRelease(new LockReleaseEvent(tableName, newTime));
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ OperatorSubtaskState state;
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().commitCount(2).build(), new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.processElement(
+ TableChange.builder().dataFileCount(1).commitCount(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ TriggerManagerOperator newOperator =
+ createOperator(
+ new TriggerEvaluator.Builder().commitCount(2).build(), new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ new OneInputStreamOperatorTestHarness<>(newOperator)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Mock a recovery trigger lock
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ testHarness.processElement(TableChange.builder().commitCount(1).build(), EVENT_TIME_2);
+
+ // Remove the lock to allow the next trigger
+ newOperator.handleOperatorEvent(lockReleaseEvent);
+ testHarness.setProcessingTime(EVENT_TIME_2);
+
+ // At this point the output contains the recovery trigger and the real trigger
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ tableName,
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new MockOperatorEventGateway(),
+ DELAY,
+ 1);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLockCheckDelay() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ tableName,
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new MockOperatorEventGateway(),
+ 1,
+ DELAY);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.open();
+
+ // Create a lock to prevent execution, and check that there is no result
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1, false);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1, false);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ operator.handleOperatorEvent(lockReleaseEvent);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManagerOperatorFactory triggerManagerOperatorFactory =
+ new TriggerManagerOperatorFactory(
+ tableName,
+ Lists.newArrayList(TASKS),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new TriggerEvaluator.Builder().commitCount(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(
+ DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), triggerManagerOperatorFactory)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitCount(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER));
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitCount(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED)))
+ .isEqualTo(1L);
+
+ // manual unlock
+ triggerManagerCoordinator.handleReleaseLock(new LockReleaseEvent(tableName, Long.MAX_VALUE));
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // manual unlock
+ triggerManagerCoordinator.handleReleaseLock(new LockReleaseEvent(tableName, Long.MAX_VALUE));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED)))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[1], "1", TRIGGERED)))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED), -1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED), -1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED), 2L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[1], "1", TRIGGERED), 1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER), 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManagerOperatorFactory manager = manager(1_000_000L, 1L);
+
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), manager)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ triggerManagerCoordinator.handleReleaseLock(lockReleaseEvent);
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED))
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManagerOperatorFactory manager = manager(1L, 1_000_000L);
+
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), manager)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED))
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED),
+ rateLimiterTrigger)
+ .put(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED),
+ concurrentRunTrigger)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED), 1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER), 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
Review Comment:
Is it used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2735375541
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
Review Comment:
Could we check that we don't accidentally overwrite any other consumer?
We use the JobManager's JVM, so we might get accidental issues if multiple job is registered on the same JM, or on job restart.
@mxm: Any ideas about the possible issues here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2736189327
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
Review Comment:
Thanks for point it out ,change it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2731171155
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockAcquireResultEvent.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * Event sent from TriggerManager operator to TableMaintenanceCoordinator to confirm that the lock
+ * has been acquired and the maintenance task is starting.
+ */
+public class LockAcquireResultEvent implements OperatorEvent {
Review Comment:
We don't need this anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2832082572
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
Review Comment:
This could be a private `record` instead of a class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2832080214
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
Review Comment:
This is a testing only method. Could we find another way to do this? Maybe create testing classes which inherit from the original, but override the `runInCoordinatorThread` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #15151:
URL: https://github.com/apache/iceberg/pull/15151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2734903211
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
+ * otherwise, it schedules the next attempt. When the job recovers from a failure, tasks from
+ * different execution paths of the previous run may still be running. Therefore, it first needs to
+ * send a lock with the maximum timestamp, and then send a recovery trigger. Only after the
+ * downstream removes this lock can we be sure that all tasks have fully stopped.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient boolean restoreTasks = false;
+ private transient boolean lockHeld = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // - fire a recovery trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ restoreTasks = true;
+ lockHeld = false;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ restoreTasks = false;
+ lockHeld = false;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleasedEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ super.processWatermark(mark);
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockHeld = false;
+ this.restoreTasks = false;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleasedEvent event) {
+ if (event.lockId().equals(tableName)) {
+ this.lockHeld = false;
+ this.restoreTasks = false;
Review Comment:
I think we can differentiate between the restoreLock release and the lockHeld release from the source from processWatermark or processElement, from processWatermark is recove. So I add a flag in LockReleasedEvent. And TriggerManagerOperator only release restoreTasksLock when it is from watermark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2735473514
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverOperator.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.iceberg.flink.maintenance.api.TaskResult;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class LockRemoverOperator extends AbstractStreamOperator<Void>
+ implements OneInputStreamOperator<TaskResult, Void>, OperatorEventHandler {
+ private static final Logger LOG = LoggerFactory.getLogger(LockRemoverOperator.class);
+
+ private static final long serialVersionUID = 1L;
+
+ private final String tableName;
+ private final OperatorEventGateway operatorEventGateway;
+ private final List<String> maintenanceTaskNames;
+ private transient List<Counter> succeededTaskResultCounters;
+ private transient List<Counter> failedTaskResultCounters;
+ private transient List<AtomicLong> taskLastRunDurationMs;
+
+ LockRemoverOperator(
+ StreamOperatorParameters<Void> parameters,
+ OperatorEventGateway operatorEventGateway,
+ String tableName,
+ List<String> maintenanceTaskNames) {
+ super(parameters);
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ }
+
+ @Override
+ public void open() throws Exception {
+ this.succeededTaskResultCounters =
+ Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.failedTaskResultCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.taskLastRunDurationMs = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ MetricGroup taskMetricGroup =
+ TableMaintenanceMetrics.groupFor(
+ getRuntimeContext(), tableName, maintenanceTaskNames.get(taskIndex), taskIndex);
+ succeededTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.SUCCEEDED_TASK_COUNTER));
+ failedTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.FAILED_TASK_COUNTER));
+ AtomicLong duration = new AtomicLong(0);
+ taskLastRunDurationMs.add(duration);
+ taskMetricGroup.gauge(TableMaintenanceMetrics.LAST_RUN_DURATION_MS, duration::get);
+ }
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ // no incoming events
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TaskResult> streamRecord) {
+ TaskResult taskResult = streamRecord.getValue();
+ LOG.info(
+ "Processing result {} for task {}",
+ taskResult,
+ maintenanceTaskNames.get(taskResult.taskIndex()));
+ long duration = System.currentTimeMillis() - taskResult.startEpoch();
+ operatorEventGateway.sendEventToCoordinator(
+ new LockReleasedEvent(tableName, streamRecord.getTimestamp()));
Review Comment:
I'm not sure about the cost of sending a message, but theoretically we could differentiate between the lock release requests simply by the lockId and the timestamp. If we keep track of the release trigger timestamp on the TriggerManager side, then we could decide if this is a recovery or a task lock.
With the current solution we always send 2 release requests when a normal task is finished. Which is unnecessary, and also might cause concurrency issues, if between the 2 release message the TriggerManager triggers another Task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2736168421
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ if (!pendingReleaseEvents.isEmpty()) {
+ pendingReleaseEvents.forEach(this::handleReleaseLock);
+ pendingReleaseEvents.clear();
+ }
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ @VisibleForTesting
+ void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ pendingReleaseEvents.add(lockReleasedEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
Review Comment:
In my understanding, some tasks are stateless and will not continue execution after a job failure and restart. If we remove the recovery lock, when restore this lock, then this lock would never be released. If I understand correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2731212420
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
+ * otherwise, it schedules the next attempt. When the job recovers from a failure, tasks from
+ * different execution paths of the previous run may still be running. Therefore, it first needs to
+ * send a lock with the maximum timestamp, and then send a recovery trigger. Only after the
+ * downstream removes this lock can we be sure that all tasks have fully stopped.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient boolean restoreTasks = false;
+ private transient boolean lockHeld = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // - fire a recovery trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ restoreTasks = true;
+ lockHeld = false;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ restoreTasks = false;
+ lockHeld = false;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleasedEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ super.processWatermark(mark);
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockHeld = false;
+ this.restoreTasks = false;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleasedEvent event) {
+ if (event.lockId().equals(tableName)) {
+ this.lockHeld = false;
+ this.restoreTasks = false;
Review Comment:
This might not be correct on recovery.
If we have an ongoing task and the recovery lock, then this might be only the "task" lock release.
Let's imagine this sequence:
- Restore with ongoing T1. We send a Trigger for restore. `restoreTasks` is set to `true`.
- T1 lock release arrives. `restoreTasks` is set to `false`.
- We trigger T2, and set `lockHeld` to `true`
- Restore Trigger arrives to the LockRemover. We receive a `LockReleasedEvent`, and set `lockHeld` to `false`.
- We trigger T3 - which is wrong
We need to differentiate between the restoreLock release and the lockHeld release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2731180337
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ private void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ LOG.info(
Review Comment:
Do we need to handle concurrency on restart?
We might get a `release` event even before the `register` event, and we would like to process them when the register even arrives
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#issuecomment-3826030562
> Thanks @Guosmilesmile! I appreciate your work on this.
>
> I think we can remove the recovery lock, if we switch to this design:
I don't really like the solution:
- It would need way more messages than a simple release lock solution
- It would mean that we need to have constant communication with the Job Manager when we are waiting for the lock
I don't think that the recovery lock is that complicated to make it worth to accept the messaging overhead and also add state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2798276568
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockReleasedEvent.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * Event sent from TriggerManagerOperatorCoordinator to TriggerManager operator to notify that a
+ * lock has been released (watermark arrived).
+ */
+public class LockReleasedEvent implements OperatorEvent {
Review Comment:
```suggestion
public class LockReleaseEvent implements OperatorEvent {
```
This is a request for releasing the lock. It hasn't been released yet.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
Review Comment:
I don't think we can assume that this message gets processed before the LockReleaseEvent. There is no ordering between the two channels between TriggerManagerOperator => TriggerManagerCoordinator |LockRemoverCoordinator <= LockRemoverOperator.
I think we need to send an ACK message to the TriggerManagerOperator before we can proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2806701417
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
Review Comment:
@pvary Thanks for the suggestion. From my perspective, we can address the out-of-order issue by synchronizing on `PENDING_RELEASE_EVENTS`, it would keep the code simpler. @mxm , could you help evaluate this as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818160992
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockTime = null;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleaseEvent event) {
+ if (lockTime == null) {
+ return;
+ }
Review Comment:
Right, we don't expect nulls here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818155490
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockReleaseEvent.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * Event sent from TriggerManagerOperatorCoordinator to TriggerManager operator to notify that a
+ * lock has been released (watermark arrived).
+ */
+public class LockReleaseEvent implements OperatorEvent {
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822364100
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManagerCoordinator.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.EventReceivingTasks;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.api.parallel.ExecutionMode;
+
+@Timeout(value = 10)
+@Execution(ExecutionMode.SAME_THREAD)
Review Comment:
Because the Coordinator uses static variables, running test cases in parallel could lead to unexpected results. In our scenario, the Coordinator won’t be executed in parallel, so I added this restriction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816075097
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManager operator to TriggerManagerCoordinator to confirm that the lock has
+ * been acquired and the maintenance task is starting.
+ */
+public class LockRegisterEvent implements OperatorEvent {
+
+ private final String lockId;
+ private final long timestamp;
Review Comment:
Do we need the timestamp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816055924
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManager operator to TriggerManagerCoordinator to confirm that the lock has
+ * been acquired and the maintenance task is starting.
Review Comment:
Is this instead registering a lock remover handler?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821241446
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ protected static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ protected static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
Review Comment:
```suggestion
public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
runInCoordinatorThread(
() -> resultFuture.complete(new byte[0]),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2828746421
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ protected static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ protected static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, @Nullable byte[] checkpointData)
+ throws Exception {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
+
+ @SuppressWarnings("unchecked")
+ protected SubtaskGateways(String operatorName, int parallelism) {
+ this.operatorName = operatorName;
+ gateways = new Map[parallelism];
+
+ for (int i = 0; i < parallelism; ++i) {
+ gateways[i] = new java.util.HashMap<>();
+ }
+ }
+
+ protected void registerSubtaskGateway(SubtaskGateway gateway) {
+ int subtaskIndex = gateway.getSubtask();
+ int attemptNumber = gateway.getExecution().getAttemptNumber();
+ Preconditions.checkState(
+ !gateways[subtaskIndex].containsKey(attemptNumber),
+ "Coordinator of %s already has a subtask gateway for %d (#%d)",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ LOG.debug(
+ "Coordinator of {} registers gateway for subtask {} attempt {}",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ gateways[subtaskIndex].put(attemptNumber, gateway);
+ LOG.debug("Registered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected void unregisterSubtaskGateway(int subtaskIndex, int attemptNumber) {
+ gateways[subtaskIndex].remove(attemptNumber);
+ LOG.debug("Unregistered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected SubtaskGateway getSubtaskGateway(int subtaskIndex) {
+ Preconditions.checkState(
+ !gateways[subtaskIndex].isEmpty(),
+ "Coordinator subtask %d is not ready yet to receive events",
+ subtaskIndex);
+ return gateways[subtaskIndex].values().iterator().next();
+ }
+
+ protected void reset(int subtaskIndex) {
+ gateways[subtaskIndex].clear();
+ }
+ }
+
+ /** Custom thread factory for the coordinator executor. */
+ protected static class CoordinatorExecutorThreadFactory
+ implements java.util.concurrent.ThreadFactory, Thread.UncaughtExceptionHandler {
Review Comment:
nit: import the `ThreadFactory` class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816252999
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestTableMaintenanceCoordinationLock.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.api;
+
+import static org.apache.iceberg.flink.SimpleDataUtil.createRowData;
+import static org.apache.iceberg.flink.maintenance.api.TableMaintenance.SOURCE_OPERATOR_NAME_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.Collections;
+import java.util.List;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.transformations.SourceTransformation;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.flink.maintenance.operator.ManualSource;
+import org.apache.iceberg.flink.maintenance.operator.OperatorTestBase;
+import org.apache.iceberg.flink.maintenance.operator.TableChange;
+import org.apache.iceberg.flink.sink.FlinkSink;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+class TestTableMaintenanceCoordinationLock extends OperatorTestBase {
+ private static final String MAINTENANCE_TASK_NAME = "TestTableMaintenance";
+ private static final String[] TASKS =
+ new String[] {MAINTENANCE_TASK_NAME + " [0]", MAINTENANCE_TASK_NAME + " [1]"};
+ private static final TableChange DUMMY_CHANGE = TableChange.builder().commitCount(1).build();
+ private static final List<Trigger> PROCESSED =
+ Collections.synchronizedList(Lists.newArrayListWithCapacity(1));
+
+ private StreamExecutionEnvironment env;
+ private Table table;
+
+ @TempDir private File checkpointDir;
+
+ @BeforeEach
+ public void beforeEach() throws IOException {
+ Configuration config = new Configuration();
+ config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
+ config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "file://" + checkpointDir.getPath());
+ this.env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ this.table = createTable();
+ insert(table, 1, "a");
+
+ PROCESSED.clear();
+ MaintenanceTaskBuilderForTest.counter = 0;
+ }
+
+ @Test
+ void testForChangeStream() throws Exception {
+ ManualSource<TableChange> schedulerSource =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+
+ TableMaintenance.Builder streamBuilder =
+ TableMaintenance.forChangeStream(schedulerSource.dataStream(), tableLoader())
+ .rateLimit(Duration.ofMillis(2))
+ .lockCheckDelay(Duration.ofSeconds(3))
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .scheduleOnDataFileCount(2)
+ .scheduleOnDataFileSize(3L)
+ .scheduleOnEqDeleteFileCount(4)
+ .scheduleOnEqDeleteRecordCount(5L)
+ .scheduleOnPosDeleteFileCount(6)
+ .scheduleOnPosDeleteRecordCount(7L)
+ .scheduleOnInterval(Duration.ofHours(1)));
+
+ sendEvents(schedulerSource, streamBuilder, ImmutableList.of(Tuple2.of(DUMMY_CHANGE, 1)));
+ }
+
+ @Test
+ void testForTable() throws Exception {
+ TableLoader tableLoader = tableLoader();
+
+ env.enableCheckpointing(10);
+
+ TableMaintenance.forTable(env, tableLoader)
+ .rateLimit(Duration.ofMillis(2))
+ .maxReadBack(2)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(2))
+ .append();
+
+ // Creating a stream for inserting data into the table concurrently
+ ManualSource<RowData> insertSource =
+ new ManualSource<>(env, InternalTypeInfo.of(FlinkSchemaUtil.convert(table.schema())));
+ FlinkSink.forRowData(insertSource.dataStream())
+ .tableLoader(tableLoader)
+ .uidPrefix(UID_SUFFIX + "-iceberg-sink")
+ .append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ insertSource.sendRecord(createRowData(2, "b"));
+
+ Awaitility.await().until(() -> PROCESSED.size() == 1);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testUidAndSlotSharingGroup() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupUnset() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, null);
+ checkSlotSharingGroupsAreSet(env, null);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupInherit() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupOverWrite() throws IOException {
+ String anotherUid = "Another-UID";
+ String anotherSlotSharingGroup = "Another-SlotSharingGroup";
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(anotherUid)
+ .slotSharingGroup(anotherSlotSharingGroup))
+ .append();
+
+ // Choose an operator from the scheduler part of the graph
+ Transformation<?> schedulerTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().equals("Trigger manager"))
+ .findFirst()
+ .orElseThrow();
+ assertThat(schedulerTransformation.getUid()).contains(UID_SUFFIX);
+ assertThat(schedulerTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(schedulerTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(SLOT_SHARING_GROUP);
+
+ // Choose an operator from the maintenance task part of the graph
+ Transformation<?> scheduledTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().startsWith(MAINTENANCE_TASK_NAME))
+ .findFirst()
+ .orElseThrow();
+ assertThat(scheduledTransformation.getUid()).contains(anotherUid);
+ assertThat(scheduledTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(scheduledTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(anotherSlotSharingGroup);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupForMonitorSource() throws IOException {
+ TableMaintenance.forTable(env, tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ Transformation<?> source = monitorSource();
+ assertThat(source).isNotNull();
+ assertThat(source.getUid()).contains(UID_SUFFIX);
+ assertThat(source.getSlotSharingGroup()).isPresent();
+ assertThat(source.getSlotSharingGroup().get().getName()).isEqualTo(SLOT_SHARING_GROUP);
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ /**
+ * Sends the events though the {@link ManualSource} provided, and waits until the given number of
+ * records are processed.
+ *
+ * @param schedulerSource used for sending the events
+ * @param streamBuilder used for generating the job
+ * @param eventsAndResultNumbers the pair of the event and the expected processed records
+ * @throws Exception if any
+ */
+ private void sendEvents(
+ ManualSource<TableChange> schedulerSource,
+ TableMaintenance.Builder streamBuilder,
+ List<Tuple2<TableChange, Integer>> eventsAndResultNumbers)
+ throws Exception {
+ streamBuilder.append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ eventsAndResultNumbers.forEach(
+ eventsAndResultNumber -> {
+ int expectedSize = PROCESSED.size() + eventsAndResultNumber.f1;
+ schedulerSource.sendRecord(eventsAndResultNumber.f0);
+ Awaitility.await().until(() -> PROCESSED.size() == expectedSize);
+ });
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ /**
+ * Finds the {@link org.apache.iceberg.flink.maintenance.operator.MonitorSource} for testing
+ * purposes by parsing the transformation tree.
+ *
+ * @return The monitor source if we found it
+ */
+ private Transformation<?> monitorSource() {
+ assertThat(env.getTransformations()).isNotEmpty();
+ assertThat(env.getTransformations().get(0).getInputs()).isNotEmpty();
+ assertThat(env.getTransformations().get(0).getInputs().get(0).getInputs()).isNotEmpty();
+
+ Transformation<?> result =
+ env.getTransformations().get(0).getInputs().get(0).getInputs().get(0);
+
+ // Some checks to make sure this is the transformation we are looking for
+ assertThat(result).isInstanceOf(SourceTransformation.class);
+ assertThat(result.getName()).startsWith(SOURCE_OPERATOR_NAME_PREFIX);
+
+ return result;
+ }
+
+ private static class MaintenanceTaskBuilderForTest
+ extends MaintenanceTaskBuilder<MaintenanceTaskBuilderForTest> {
+ private final boolean success;
+ private final int id;
+ private static int counter = 0;
+
+ MaintenanceTaskBuilderForTest(boolean success) {
+ this.success = success;
+ this.id = counter;
+ ++counter;
+ }
+
+ @Override
+ String maintenanceTaskName() {
+ return MAINTENANCE_TASK_NAME;
+ }
+
+ @Override
+ DataStream<TaskResult> append(DataStream<Trigger> trigger) {
+ String name = TASKS[id];
+ return trigger
+ .map(new DummyMaintenanceTask(success))
+ .name(name)
+ .uid(uidSuffix() + "-test-mapper-" + name + "-" + id)
+ .slotSharingGroup(slotSharingGroup())
+ .forceNonParallel();
+ }
+ }
+
+ private static class DummyMaintenanceTask
+ implements MapFunction<Trigger, TaskResult>, ResultTypeQueryable<TaskResult>, Serializable {
+ private final boolean success;
+
+ private DummyMaintenanceTask(boolean success) {
+ this.success = success;
+ }
+
+ @Override
+ public TaskResult map(Trigger trigger) {
+ // Ensure that the lock is held when processing
+ // assertThat(LOCK_FACTORY.createLock().isHeld()).isTrue();
Review Comment:
nit: a bit of strange formatting. Delete whitespaces and allow spotless to work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2812002524
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
Review Comment:
Synchronization wouldn't solve this issue. This is about the order at which the messages are processed in independent channels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2764296961
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private final List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ pendingReleaseEvents.forEach(this::handleReleaseLock);
+ pendingReleaseEvents.clear();
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ @VisibleForTesting
+ void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ pendingReleaseEvents.add(lockReleasedEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {}
Review Comment:
Your are right, I have add a clear in `resetToCheckpoint`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816047747
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManager operator to TriggerManagerCoordinator to confirm that the lock has
+ * been acquired and the maintenance task is starting.
+ */
+public class LockRegisterEvent implements OperatorEvent {
+
+ private final String lockId;
+ private final long timestamp;
+
+ public LockRegisterEvent(String lockId, long timestamp) {
+ this.lockId = lockId;
+ this.timestamp = timestamp;
+ }
+
+ public String lockId() {
+ return lockId;
+ }
+
+ public long timestamp() {
+ return timestamp;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (!(o instanceof LockRegisterEvent)) {
+ return false;
+ }
+
+ LockRegisterEvent that = (LockRegisterEvent) o;
+ return Objects.equal(timestamp, that.timestamp) && Objects.equal(lockId, that.lockId);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(timestamp, lockId);
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("timeStamp", timestamp)
Review Comment:
nit: `timestamp`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816068635
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockReleaseEvent.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * Event sent from TriggerManagerOperatorCoordinator to TriggerManager operator to notify that a
+ * lock has been released (watermark arrived).
Review Comment:
This is sent from the LockRemoverOperator to the Coordinator, and then from the TMCoordinator to the TM about the lock release. I assume that the 2 events are the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2832087851
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
Review Comment:
Do we need to have an array here? Both LockRemover and TriggerManager is a non-parallel task. We could just use Preconditions to make sure that this not changes, and handle only a single subtask
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2832067897
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
Review Comment:
Do we need these?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2832066166
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
Review Comment:
Not only for testing anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2832091152
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
+
+ @SuppressWarnings("unchecked")
+ protected SubtaskGateways(String operatorName, int parallelism) {
+ this.operatorName = operatorName;
+ gateways = new Map[parallelism];
+
+ for (int i = 0; i < parallelism; ++i) {
+ gateways[i] = new java.util.HashMap<>();
+ }
+ }
+
+ protected void registerSubtaskGateway(SubtaskGateway gateway) {
+ int subtaskIndex = gateway.getSubtask();
+ int attemptNumber = gateway.getExecution().getAttemptNumber();
+ Preconditions.checkState(
+ !gateways[subtaskIndex].containsKey(attemptNumber),
+ "Coordinator of %s already has a subtask gateway for %d (#%d)",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ LOG.debug(
+ "Coordinator of {} registers gateway for subtask {} attempt {}",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ gateways[subtaskIndex].put(attemptNumber, gateway);
+ LOG.debug("Registered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected void unregisterSubtaskGateway(int subtaskIndex, int attemptNumber) {
+ gateways[subtaskIndex].remove(attemptNumber);
+ LOG.debug("Unregistered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected SubtaskGateway getSubtaskGateway(int subtaskIndex) {
+ Preconditions.checkState(
+ !gateways[subtaskIndex].isEmpty(),
+ "Coordinator subtask %d is not ready yet to receive events",
+ subtaskIndex);
+ return gateways[subtaskIndex].values().iterator().next();
Review Comment:
Is this correct? we ignore the attempt number when getting the gateway?
If we do so, do we need to store the attempt number at all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2836150305
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
Review Comment:
I tried to integrate a concrete Coordinator in the tests and override `runInCoordinatorThread`, but `executionAttemptReady` and other methods require checking `isCurrentThreadCoordinatorThread`, which would cause failures.
To move forward, we’d need to add an accessor for the `coordinatorThreadFactory`, and `CoordinatorExecutorThreadFactory` would also need to expose more methods. So I kept the current implementation as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821302025
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockRelease((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockTime = null;
+ }
+
+ @VisibleForTesting
+ void handleLockRelease(LockReleaseEvent event) {
+ Preconditions.checkArgument(lockTime != null, "Lock time is null, Can't release lock");
+
+ if (event.timestamp() >= lockTime) {
+ this.lockTime = null;
+ this.shouldRestoreTasks = false;
+ }
+ }
+
+ private void checkAndFire(ProcessingTimeService timerService) {
+ long current = timerService.getCurrentProcessingTime();
+ if (shouldRestoreTasks) {
+ if (lockTime != null) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.info("The recovery lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ }
+ }
+
+ Integer taskToStart =
+ TriggerUtil.nextTrigger(
+ evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (!triggered) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ } else {
+ LOG.debug("Execution check finished");
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ triggered = false;
+ return;
+ }
+
+ if (lockTime == null) {
+ this.lockTime = current;
+ TableChange change = accumulatedChanges.get(taskToStart);
+ output.collect(new StreamRecord<>(Trigger.create(current, taskToStart), current));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, tableName);
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = (taskToStart + 1) % evaluators.size();
+ triggered = true;
+ } else {
+ // A task is already running, waiting for it to finish
+ LOG.info("Failed to acquire lock. Delaying task to {}", current + lockCheckDelayMs);
+
+ startsFrom = taskToStart;
+ concurrentRunThrottledCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
Review Comment:
Do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821303993
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockRelease((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
Review Comment:
Do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822385184
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManagerOperator to TriggerManagerCoordinator to register a lock release
+ * handler. This handler will be used to forward lock release events back to the operator when
+ * triggered by downstream operators.
+ */
+public class LockRegisterEvent implements OperatorEvent {
+
+ private final String lockId;
+
+ public LockRegisterEvent(String lockId) {
+ this.lockId = lockId;
+ }
+
+ public String lockId() {
+ return lockId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
Review Comment:
Originally, this class was used as the key in a map. It’s no longer needed, so I removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821250107
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ protected static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ protected static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
Review Comment:
Do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816027393
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -81,11 +81,21 @@ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
lockReleaseEvent.lockId(),
lockReleaseEvent.timestamp());
} else {
Review Comment:
As you mentioned above, I may have overcomplicated things here by introducing premature optimizations, so I’ve adjusted the code in this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816010968
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
Review Comment:
Synchronized methods are like:
```
synchronized(this) {
[..]
}
```
We will have 2 different coordinator objects. We need to sync on the static variable which is shared
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2734904236
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ private void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ LOG.info(
Review Comment:
I add a pendingReleaseEvents in this case , and when we registerTriggerManagerReceiveReleaseEvent will replay the pendingReleaseEvents again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2734911026
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverOperator.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.iceberg.flink.maintenance.api.TaskResult;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class LockRemoverOperator extends AbstractStreamOperator<Void>
+ implements OneInputStreamOperator<TaskResult, Void>, OperatorEventHandler {
+ private static final Logger LOG = LoggerFactory.getLogger(LockRemoverOperator.class);
+
+ private static final long serialVersionUID = 1L;
+
+ private final String tableName;
+ private final OperatorEventGateway operatorEventGateway;
+ private final List<String> maintenanceTaskNames;
+ private transient List<Counter> succeededTaskResultCounters;
+ private transient List<Counter> failedTaskResultCounters;
+ private transient List<AtomicLong> taskLastRunDurationMs;
+
+ LockRemoverOperator(
+ StreamOperatorParameters<Void> parameters,
+ OperatorEventGateway operatorEventGateway,
+ String tableName,
+ List<String> maintenanceTaskNames) {
+ super(parameters);
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ }
+
+ @Override
+ public void open() throws Exception {
+ this.succeededTaskResultCounters =
+ Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.failedTaskResultCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.taskLastRunDurationMs = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ MetricGroup taskMetricGroup =
+ TableMaintenanceMetrics.groupFor(
+ getRuntimeContext(), tableName, maintenanceTaskNames.get(taskIndex), taskIndex);
+ succeededTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.SUCCEEDED_TASK_COUNTER));
+ failedTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.FAILED_TASK_COUNTER));
+ AtomicLong duration = new AtomicLong(0);
+ taskLastRunDurationMs.add(duration);
+ taskMetricGroup.gauge(TableMaintenanceMetrics.LAST_RUN_DURATION_MS, duration::get);
+ }
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ // no incoming events
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TaskResult> streamRecord) {
+ TaskResult taskResult = streamRecord.getValue();
+ LOG.info(
+ "Processing result {} for task {}",
+ taskResult,
+ maintenanceTaskNames.get(taskResult.taskIndex()));
+ long duration = System.currentTimeMillis() - taskResult.startEpoch();
+ operatorEventGateway.sendEventToCoordinator(
+ new LockReleasedEvent(tableName, streamRecord.getTimestamp()));
Review Comment:
Now I keep the code since the reason above I use processWatermark or processElement to differentiate between the restoreLock release and the lockHeld release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2764309133
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
Review Comment:
I add a clear in resetToCheckpoint, keep only a few lock in the coordinator map
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile closed pull request #15151: Flink: TableMaintenance Support Coordinator Lock
URL: https://github.com/apache/iceberg/pull/15151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2812284970
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
Review Comment:
Never mind, synchronization does solve the issue, as does adding an ack message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2741188867
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
+ * otherwise, it schedules the next attempt. When the job recovers from a failure, tasks from
+ * different execution paths of the previous run may still be running. Therefore, it first needs to
+ * send a lock with the maximum timestamp, and then send a recovery trigger. Only after the
+ * downstream removes this lock can we be sure that all tasks have fully stopped.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // - fire a recovery trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
Review Comment:
There is no tryLock here.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ if (!pendingReleaseEvents.isEmpty()) {
+ pendingReleaseEvents.forEach(this::handleReleaseLock);
+ pendingReleaseEvents.clear();
+ }
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ @VisibleForTesting
+ void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ pendingReleaseEvents.add(lockReleasedEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
Review Comment:
Stateless tasks who do not continue after restore, should send a lock release message on restore. This will ensure that the lock will get released if it was previously held.
In general, if we want table maintenance to be stateful, the lock state should be included as well. We have the option to do this now because we maintain the lock state here.
See also https://github.com/apache/iceberg/pull/15042#discussion_r2741218393.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
Review Comment:
AFAIK this is no longer true for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2734901083
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockAcquireResultEvent.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * Event sent from TriggerManager operator to TableMaintenanceCoordinator to confirm that the lock
+ * has been acquired and the maintenance task is starting.
+ */
+public class LockAcquireResultEvent implements OperatorEvent {
Review Comment:
You are right, remove it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2735856002
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
Review Comment:
Have we considered a different flow, where we confirm the lock registration? Why are we assuming that every lock register event succeeds?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private final List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName);
+ this.subtaskGateways.getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ pendingReleaseEvents.forEach(this::handleReleaseLock);
+ pendingReleaseEvents.clear();
+ }
+
+ /** Release the lock and optionally trigger the next pending task. */
+ @VisibleForTesting
+ void handleReleaseLock(LockReleasedEvent lockReleasedEvent) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleasedEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleasedEvent.lockId()).accept(lockReleasedEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ } else {
+ pendingReleaseEvents.add(lockReleasedEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleasedEvent.lockId(),
+ lockReleasedEvent.timestamp());
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {}
Review Comment:
Should we clear the map here and let the operators re-acquire the lock?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
Review Comment:
Considering that this map will never be cleared, is there a risk that we might leak memory if we run table maintenance on many tables?
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private boolean started;
+ private final transient SubtaskGateways subtaskGateways;
Review Comment:
```suggestion
private final SubtaskGateways subtaskGateways;
```
Not really transient.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
Review Comment:
Would it be possible to break this into a base class and two implementation, one coordinator class for the TriggerManagerOperator and one for the LockRemoverOperator? I think that would greatly help understanding how the code works and provide a better separation of concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816228680
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockTime = null;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleaseEvent event) {
+ if (lockTime == null) {
+ return;
+ }
Review Comment:
Is this a Precondition? I don't think we expect nulls here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816206923
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
Review Comment:
Don't need both. Either internal, or experimental.
Experimental if it is called by external users - I expect it is not needed here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816024008
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
Review Comment:
Change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818139127
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRegisterEvent.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+
+/**
+ * Event sent from TriggerManager operator to TriggerManagerCoordinator to confirm that the lock has
+ * been acquired and the maintenance task is starting.
+ */
+public class LockRegisterEvent implements OperatorEvent {
+
+ private final String lockId;
+ private final long timestamp;
Review Comment:
Remove it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816039700
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ protected static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ protected static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS =
+ new CopyOnWriteArrayList<>();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
Review Comment:
Why do we have the `handleReleaseLock` in the Base, but the register in the TMCoordinator?
My first instinct would be have the `handleReleaseLock` in the LRCoordinator, and the `registerLock` (which maybe should be `handleRegisterLock` for consistency) in the TMCoordinator.
My second thought is that maybe we should keep both of them here, because of the locking magic, so the logic is in one place. I would leave it to you guys, but please be consistent 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816123203
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockReleaseEvent.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * Event sent from TriggerManagerOperatorCoordinator to TriggerManager operator to notify that a
+ * lock has been released (watermark arrived).
+ */
+public class LockReleaseEvent implements OperatorEvent {
Review Comment:
Annotation of the classes please `@Internal`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821358683
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTriggerManagerOperator.java:
##########
@@ -0,0 +1,679 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.NOTHING_TO_TRIGGER;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED;
+import static org.apache.iceberg.flink.maintenance.operator.TableMaintenanceMetrics.TRIGGERED;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.operators.coordination.MockOperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+class TestTriggerManagerOperator extends OperatorTestBase {
+ private static final long DELAY = 10L;
+ private static final String OPERATOR_NAME = "TestCoordinator";
+ private static final OperatorID TEST_OPERATOR_ID = new OperatorID(1234L, 5678L);
+ private static final String[] TASKS = new String[] {"task0", "task1"};
+ private long processingTime = 0L;
+ private String tableName;
+ private TriggerManagerCoordinator triggerManagerCoordinator;
+ private LockReleaseEvent lockReleaseEvent;
+
+ @BeforeEach
+ void before() {
+ super.before();
+ Table table = createTable();
+ this.tableName = table.name();
+ lockReleaseEvent = new LockReleaseEvent(tableName, 1L);
+ this.triggerManagerCoordinator = createCoordinator();
+ try {
+ triggerManagerCoordinator.start();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @AfterEach
+ void after() throws IOException {
+ super.after();
+ try {
+ triggerManagerCoordinator.close();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ void testCommitCount() throws Exception {
+ MockOperatorEventGateway mockGateway = new MockOperatorEventGateway();
+ TriggerManagerOperator operator =
+ createOperator(new TriggerEvaluator.Builder().commitCount(3).build(), mockGateway);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testDataFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().dataFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(1).build(), 0);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(5).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileCount(2).build(), 4);
+ }
+ }
+
+ @Test
+ void testDataFileSizeInBytes() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().dataFileSizeInBytes(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().dataFileSizeInBytes(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testPosDeleteFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().posDeleteFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteFileCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testPosDeleteRecordCount() throws Exception {
+
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().posDeleteRecordCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().posDeleteRecordCount(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testEqDeleteFileCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().eqDeleteFileCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(2).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(3).build(), 2);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(10).build(), 3);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 3);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 3);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteFileCount(1).build(), 4);
+ }
+ }
+
+ @Test
+ void testEqDeleteRecordCount() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().eqDeleteRecordCount(3).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(1L).build(), 0);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(2L).build(), 1);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(5L).build(), 2);
+
+ // No trigger in this case
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(1L).build(), 2);
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().eqDeleteRecordCount(2L).build(), 3);
+ }
+ }
+
+ @Test
+ void testTimeout() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().timeout(Duration.ofSeconds(1)).build(),
+ new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ TableChange event = TableChange.builder().dataFileCount(1).commitCount(1).build();
+
+ // Wait for some time
+ testHarness.processElement(event, EVENT_TIME);
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ // Wait for the timeout to expire
+ long newTime = EVENT_TIME + Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Remove the lock to allow the next trigger
+ operator.handleLockRelease(new LockReleaseEvent(tableName, newTime));
+
+ // Send a new event
+ testHarness.setProcessingTime(newTime + 1);
+ testHarness.processElement(event, newTime);
+
+ // No trigger yet
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Send a new event
+ newTime += Duration.ofSeconds(1).toMillis();
+ testHarness.setProcessingTime(newTime);
+ testHarness.processElement(event, newTime);
+
+ // New trigger should arrive
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testStateRestore() throws Exception {
+ OperatorSubtaskState state;
+ TriggerManagerOperator operator =
+ createOperator(
+ new TriggerEvaluator.Builder().commitCount(2).build(), new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.processElement(
+ TableChange.builder().dataFileCount(1).commitCount(1).build(), EVENT_TIME);
+
+ assertThat(testHarness.extractOutputValues()).isEmpty();
+
+ state = testHarness.snapshot(1, EVENT_TIME);
+ }
+
+ // Restore the state, write some more data, create a checkpoint, check the data which is written
+ TriggerManagerOperator newOperator =
+ createOperator(
+ new TriggerEvaluator.Builder().commitCount(2).build(), new MockOperatorEventGateway());
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ new OneInputStreamOperatorTestHarness<>(newOperator)) {
+ testHarness.initializeState(state);
+ testHarness.open();
+
+ // Mock a recovery trigger lock
+ assertTriggers(
+ testHarness.extractOutputValues(),
+ Lists.newArrayList(Trigger.recovery(testHarness.getProcessingTime())));
+
+ testHarness.processElement(TableChange.builder().commitCount(1).build(), EVENT_TIME_2);
+
+ // Remove the lock to allow the next trigger
+ newOperator.handleOperatorEvent(lockReleaseEvent);
+ testHarness.setProcessingTime(EVENT_TIME_2);
+
+ // At this point the output contains the recovery trigger and the real trigger
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testMinFireDelay() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ tableName,
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new MockOperatorEventGateway(),
+ DELAY,
+ 1);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.open();
+
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+ long currentTime = testHarness.getProcessingTime();
+
+ // No new fire yet
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testLockCheckDelay() throws Exception {
+ TriggerManagerOperator operator =
+ createOperator(
+ tableName,
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new MockOperatorEventGateway(),
+ 1,
+ DELAY);
+ try (OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness =
+ createHarness(operator)) {
+ testHarness.open();
+
+ // Create a lock to prevent execution, and check that there is no result
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1, false);
+ addEventAndCheckResult(
+ operator, testHarness, TableChange.builder().commitCount(2).build(), 1, false);
+ long currentTime = testHarness.getProcessingTime();
+
+ // Remove the lock, and still no trigger
+ operator.handleOperatorEvent(lockReleaseEvent);
+ assertThat(testHarness.extractOutputValues()).hasSize(1);
+
+ // Check that the trigger fired after the delay
+ testHarness.setProcessingTime(currentTime + DELAY);
+ assertThat(testHarness.extractOutputValues()).hasSize(2);
+ }
+ }
+
+ @Test
+ void testTriggerMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ TriggerManagerOperatorFactory triggerManagerOperatorFactory =
+ new TriggerManagerOperatorFactory(
+ tableName,
+ Lists.newArrayList(TASKS),
+ Lists.newArrayList(
+ new TriggerEvaluator.Builder().commitCount(2).build(),
+ new TriggerEvaluator.Builder().commitCount(4).build()),
+ 1L,
+ 1L);
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(
+ DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), triggerManagerOperatorFactory)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // This one doesn't trigger - tests NOTHING_TO_TRIGGER
+ source.sendRecord(TableChange.builder().commitCount(1).build());
+
+ Awaitility.await()
+ .until(
+ () -> {
+ Long notingCounter =
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER));
+ return notingCounter != null && notingCounter.equals(1L);
+ });
+
+ // Trigger one of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitCount(1).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED)))
+ .isEqualTo(1L);
+
+ // manual unlock
+ triggerManagerCoordinator.handleReleaseLock(new LockReleaseEvent(tableName, Long.MAX_VALUE));
+ // Trigger both of the tasks - tests TRIGGERED
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ // Wait until we receive the trigger
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // manual unlock
+ triggerManagerCoordinator.handleReleaseLock(new LockReleaseEvent(tableName, Long.MAX_VALUE));
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED)))
+ .isEqualTo(2L);
+ assertThat(
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[1], "1", TRIGGERED)))
+ .isEqualTo(1L);
+
+ // Final check all the counters
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED), -1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED), -1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED), 2L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[1], "1", TRIGGERED), 1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER), 1L)
+ .build());
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testRateLimiterMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManagerOperatorFactory manager = manager(1_000_000L, 1L);
+
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), manager)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // Remove the lock to allow the next trigger
+ triggerManagerCoordinator.handleReleaseLock(lockReleaseEvent);
+
+ // The second trigger will be blocked
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED))
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(1L, 0L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testConcurrentRunMetrics() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ ManualSource<TableChange> source =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+ CollectingSink<Trigger> sink = new CollectingSink<>();
+
+ // High delay, so only triggered once
+ TriggerManagerOperatorFactory manager = manager(1L, 1_000_000L);
+
+ source
+ .dataStream()
+ .keyBy(unused -> true)
+ .transform(DUMMY_TASK_NAME, TypeInformation.of(Trigger.class), manager)
+ .forceNonParallel()
+ .sinkTo(sink);
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ // Start the first trigger - notice that we do not remove the lock after the trigger
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ assertThat(sink.poll(Duration.ofSeconds(5))).isNotNull();
+
+ // The second trigger will be blocked by the lock
+ source.sendRecord(TableChange.builder().commitCount(2).build());
+ Awaitility.await()
+ .until(
+ () ->
+ MetricsReporterFactoryForTests.counter(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED))
+ .equals(1L));
+
+ // Final check all the counters
+ assertCounters(0L, 1L);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ private void assertCounters(long rateLimiterTrigger, long concurrentRunTrigger) {
+ MetricsReporterFactoryForTests.assertCounters(
+ new ImmutableMap.Builder<List<String>, Long>()
+ .put(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, RATE_LIMITER_TRIGGERED),
+ rateLimiterTrigger)
+ .put(
+ ImmutableList.of(DUMMY_TASK_NAME, tableName, CONCURRENT_RUN_THROTTLED),
+ concurrentRunTrigger)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, TASKS[0], "0", TRIGGERED), 1L)
+ .put(ImmutableList.of(DUMMY_TASK_NAME, tableName, NOTHING_TO_TRIGGER), 0L)
+ .build());
+ }
+
+ private KeyedOneInputStreamOperatorTestHarness<Boolean, TableChange, Trigger> harness(
+ TriggerManager manager) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new KeyedProcessOperator<>(manager), value -> true, Types.BOOLEAN);
+ }
+
+ private void addEventAndCheckResult(
+ TriggerManagerOperator operator,
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize)
+ throws Exception {
+ addEventAndCheckResult(operator, testHarness, event, expectedSize, true);
+ }
+
+ private void addEventAndCheckResult(
+ TriggerManagerOperator operator,
+ OneInputStreamOperatorTestHarness<TableChange, Trigger> testHarness,
+ TableChange event,
+ int expectedSize,
+ boolean removeLock)
+ throws Exception {
+ ++processingTime;
+ testHarness.setProcessingTime(processingTime);
+ testHarness.processElement(event, processingTime);
+ assertThat(testHarness.extractOutputValues()).hasSize(expectedSize);
+ if (removeLock && operator.lockTime() != null) {
+ // Remove the lock to allow the next trigger
+ operator.handleLockRelease(new LockReleaseEvent(tableName, processingTime));
+ }
+ }
+
+ private TriggerManagerOperatorFactory manager(long minFireDelayMs, long lockCheckDelayMs) {
+ return new TriggerManagerOperatorFactory(
+ tableName,
+ Lists.newArrayList(TASKS[0]),
+ Lists.newArrayList(new TriggerEvaluator.Builder().commitCount(2).build()),
+ minFireDelayMs,
+ lockCheckDelayMs);
+ }
+
+ private static void assertTriggers(List<Trigger> expected, List<Trigger> actual) {
+ assertThat(actual).hasSize(expected.size());
+ for (int i = 0; i < expected.size(); ++i) {
+ Trigger expectedTrigger = expected.get(i);
+ Trigger actualTrigger = actual.get(i);
+ assertThat(actualTrigger.timestamp()).isEqualTo(expectedTrigger.timestamp());
+ assertThat(actualTrigger.taskId()).isEqualTo(expectedTrigger.taskId());
+ assertThat(actualTrigger.isRecovery()).isEqualTo(expectedTrigger.isRecovery());
+ }
+ }
+
+ private static TriggerManagerCoordinator createCoordinator() {
+ return new TriggerManagerCoordinator(
+ OPERATOR_NAME, new MockOperatorCoordinatorContext(TEST_OPERATOR_ID, 2));
+ }
+
+ private TriggerManagerOperator createOperator(
+ TriggerEvaluator evaluator, OperatorEventGateway mockGateway) {
+ return createOperator(tableName, evaluator, mockGateway, 1, 1);
+ }
+
+ private TriggerManagerOperator createOperator(
+ String lockId,
+ TriggerEvaluator evaluator,
+ OperatorEventGateway mockGateway,
+ long minFireDelayMs,
+ long lockCheckDelayMs) {
+ TriggerManagerOperator operator =
+ new TriggerManagerOperator(
+ null,
+ mockGateway,
+ Lists.newArrayList(TASKS[0]),
+ Lists.newArrayList(evaluator),
+ minFireDelayMs,
+ lockCheckDelayMs,
+ lockId);
+ return operator;
Review Comment:
```suggestion
return new TriggerManagerOperator(
null,
mockGateway,
Lists.newArrayList(TASKS[0]),
Lists.newArrayList(evaluator),
minFireDelayMs,
lockCheckDelayMs,
lockId);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821329454
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/api/TestTableMaintenanceCoordinationLock.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.api;
+
+import static org.apache.iceberg.flink.SimpleDataUtil.createRowData;
+import static org.apache.iceberg.flink.maintenance.api.TableMaintenance.SOURCE_OPERATOR_NAME_PREFIX;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.time.Duration;
+import java.util.Collections;
+import java.util.List;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.transformations.SourceTransformation;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.TableLoader;
+import org.apache.iceberg.flink.maintenance.operator.ManualSource;
+import org.apache.iceberg.flink.maintenance.operator.OperatorTestBase;
+import org.apache.iceberg.flink.maintenance.operator.TableChange;
+import org.apache.iceberg.flink.sink.FlinkSink;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+class TestTableMaintenanceCoordinationLock extends OperatorTestBase {
+ private static final String MAINTENANCE_TASK_NAME = "TestTableMaintenance";
+ private static final String[] TASKS =
+ new String[] {MAINTENANCE_TASK_NAME + " [0]", MAINTENANCE_TASK_NAME + " [1]"};
+ private static final TableChange DUMMY_CHANGE = TableChange.builder().commitCount(1).build();
+ private static final List<Trigger> PROCESSED =
+ Collections.synchronizedList(Lists.newArrayListWithCapacity(1));
+
+ private StreamExecutionEnvironment env;
+ private Table table;
+
+ @TempDir private File checkpointDir;
+
+ @BeforeEach
+ public void beforeEach() throws IOException {
+ Configuration config = new Configuration();
+ config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
+ config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "file://" + checkpointDir.getPath());
+ this.env = StreamExecutionEnvironment.getExecutionEnvironment(config);
+ this.table = createTable();
+ insert(table, 1, "a");
+
+ PROCESSED.clear();
+ MaintenanceTaskBuilderForTest.counter = 0;
+ }
+
+ @Test
+ void testForChangeStream() throws Exception {
+ ManualSource<TableChange> schedulerSource =
+ new ManualSource<>(env, TypeInformation.of(TableChange.class));
+
+ TableMaintenance.Builder streamBuilder =
+ TableMaintenance.forChangeStream(schedulerSource.dataStream(), tableLoader())
+ .rateLimit(Duration.ofMillis(2))
+ .lockCheckDelay(Duration.ofSeconds(3))
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .scheduleOnDataFileCount(2)
+ .scheduleOnDataFileSize(3L)
+ .scheduleOnEqDeleteFileCount(4)
+ .scheduleOnEqDeleteRecordCount(5L)
+ .scheduleOnPosDeleteFileCount(6)
+ .scheduleOnPosDeleteRecordCount(7L)
+ .scheduleOnInterval(Duration.ofHours(1)));
+
+ sendEvents(schedulerSource, streamBuilder, ImmutableList.of(Tuple2.of(DUMMY_CHANGE, 1)));
+ }
+
+ @Test
+ void testForTable() throws Exception {
+ TableLoader tableLoader = tableLoader();
+
+ env.enableCheckpointing(10);
+
+ TableMaintenance.forTable(env, tableLoader)
+ .rateLimit(Duration.ofMillis(2))
+ .maxReadBack(2)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(2))
+ .append();
+
+ // Creating a stream for inserting data into the table concurrently
+ ManualSource<RowData> insertSource =
+ new ManualSource<>(env, InternalTypeInfo.of(FlinkSchemaUtil.convert(table.schema())));
+ FlinkSink.forRowData(insertSource.dataStream())
+ .tableLoader(tableLoader)
+ .uidPrefix(UID_SUFFIX + "-iceberg-sink")
+ .append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ insertSource.sendRecord(createRowData(2, "b"));
+
+ Awaitility.await().until(() -> PROCESSED.size() == 1);
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ @Test
+ void testUidAndSlotSharingGroup() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupUnset() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, null);
+ checkSlotSharingGroupsAreSet(env, null);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupInherit() throws IOException {
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(new MaintenanceTaskBuilderForTest(true).scheduleOnCommitCount(1))
+ .append();
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupOverWrite() throws IOException {
+ String anotherUid = "Another-UID";
+ String anotherSlotSharingGroup = "Another-SlotSharingGroup";
+ TableMaintenance.forChangeStream(
+ new ManualSource<>(env, TypeInformation.of(TableChange.class)).dataStream(),
+ tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(anotherUid)
+ .slotSharingGroup(anotherSlotSharingGroup))
+ .append();
+
+ // Choose an operator from the scheduler part of the graph
+ Transformation<?> schedulerTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().equals("Trigger manager"))
+ .findFirst()
+ .orElseThrow();
+ assertThat(schedulerTransformation.getUid()).contains(UID_SUFFIX);
+ assertThat(schedulerTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(schedulerTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(SLOT_SHARING_GROUP);
+
+ // Choose an operator from the maintenance task part of the graph
+ Transformation<?> scheduledTransformation =
+ env.getTransformations().stream()
+ .filter(t -> t.getName().startsWith(MAINTENANCE_TASK_NAME))
+ .findFirst()
+ .orElseThrow();
+ assertThat(scheduledTransformation.getUid()).contains(anotherUid);
+ assertThat(scheduledTransformation.getSlotSharingGroup()).isPresent();
+ assertThat(scheduledTransformation.getSlotSharingGroup().get().getName())
+ .isEqualTo(anotherSlotSharingGroup);
+ }
+
+ @Test
+ void testUidAndSlotSharingGroupForMonitorSource() throws IOException {
+ TableMaintenance.forTable(env, tableLoader())
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP)
+ .add(
+ new MaintenanceTaskBuilderForTest(true)
+ .scheduleOnCommitCount(1)
+ .uidSuffix(UID_SUFFIX)
+ .slotSharingGroup(SLOT_SHARING_GROUP))
+ .append();
+
+ Transformation<?> source = monitorSource();
+ assertThat(source).isNotNull();
+ assertThat(source.getUid()).contains(UID_SUFFIX);
+ assertThat(source.getSlotSharingGroup()).isPresent();
+ assertThat(source.getSlotSharingGroup().get().getName()).isEqualTo(SLOT_SHARING_GROUP);
+
+ checkUidsAreSet(env, UID_SUFFIX);
+ checkSlotSharingGroupsAreSet(env, SLOT_SHARING_GROUP);
+ }
+
+ /**
+ * Sends the events though the {@link ManualSource} provided, and waits until the given number of
+ * records are processed.
+ *
+ * @param schedulerSource used for sending the events
+ * @param streamBuilder used for generating the job
+ * @param eventsAndResultNumbers the pair of the event and the expected processed records
+ * @throws Exception if any
+ */
+ private void sendEvents(
+ ManualSource<TableChange> schedulerSource,
+ TableMaintenance.Builder streamBuilder,
+ List<Tuple2<TableChange, Integer>> eventsAndResultNumbers)
+ throws Exception {
+ streamBuilder.append();
+
+ JobClient jobClient = null;
+ try {
+ jobClient = env.executeAsync();
+
+ eventsAndResultNumbers.forEach(
+ eventsAndResultNumber -> {
+ int expectedSize = PROCESSED.size() + eventsAndResultNumber.f1;
+ schedulerSource.sendRecord(eventsAndResultNumber.f0);
+ Awaitility.await().until(() -> PROCESSED.size() == expectedSize);
+ });
+ } finally {
+ closeJobClient(jobClient);
+ }
+ }
+
+ /**
+ * Finds the {@link org.apache.iceberg.flink.maintenance.operator.MonitorSource} for testing
+ * purposes by parsing the transformation tree.
+ *
+ * @return The monitor source if we found it
+ */
+ private Transformation<?> monitorSource() {
Review Comment:
Is it worth to reuse/refactor `MaintenanceTaskInfraExtension`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2731193394
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
+ * otherwise, it schedules the next attempt. When the job recovers from a failure, tasks from
+ * different execution paths of the previous run may still be running. Therefore, it first needs to
+ * send a lock with the maximum timestamp, and then send a recovery trigger. Only after the
+ * downstream removes this lock can we be sure that all tasks have fully stopped.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient boolean restoreTasks = false;
+ private transient boolean lockHeld = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // - fire a recovery trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ restoreTasks = true;
+ lockHeld = false;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ restoreTasks = false;
+ lockHeld = false;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleasedEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ super.processWatermark(mark);
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockHeld = false;
+ this.restoreTasks = false;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleasedEvent event) {
+ if (event.lockId().equals(tableName)) {
Review Comment:
Do we need this check, or the filtering could be handled in the coordinator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818158580
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinatorProvider.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator;
+
+@Internal
+public class TriggerManagerCoordinatorProvider extends RecreateOnResetOperatorCoordinator.Provider {
Review Comment:
Yes ,the same above.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockTime = null;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleaseEvent event) {
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2818154103
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverCoordinator.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Coordinator for LockRemoverOperator. Handles lock release events from downstream operators. */
+@Experimental
+@Internal
+public class LockRemoverCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(LockRemoverCoordinator.class);
+
+ public LockRemoverCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created LockRemoverCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockReleaseEvent) {
+ handleReleaseLock((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
Review Comment:
Yes, move it to the base.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverCoordinator.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Coordinator for LockRemoverOperator. Handles lock release events from downstream operators. */
+@Experimental
+@Internal
+public class LockRemoverCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(LockRemoverCoordinator.class);
+
+ public LockRemoverCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created LockRemoverCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockReleaseEvent) {
+ handleReleaseLock((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, @Nullable byte[] checkpointData)
+ throws Exception {}
Review Comment:
Yes , move it to the base
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2832093060
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
+
+ @SuppressWarnings("unchecked")
+ protected SubtaskGateways(String operatorName, int parallelism) {
+ this.operatorName = operatorName;
+ gateways = new Map[parallelism];
+
+ for (int i = 0; i < parallelism; ++i) {
+ gateways[i] = new java.util.HashMap<>();
+ }
+ }
+
+ protected void registerSubtaskGateway(SubtaskGateway gateway) {
+ int subtaskIndex = gateway.getSubtask();
+ int attemptNumber = gateway.getExecution().getAttemptNumber();
+ Preconditions.checkState(
+ !gateways[subtaskIndex].containsKey(attemptNumber),
+ "Coordinator of %s already has a subtask gateway for %d (#%d)",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ LOG.debug(
+ "Coordinator of {} registers gateway for subtask {} attempt {}",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ gateways[subtaskIndex].put(attemptNumber, gateway);
+ LOG.debug("Registered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected void unregisterSubtaskGateway(int subtaskIndex, int attemptNumber) {
+ gateways[subtaskIndex].remove(attemptNumber);
+ LOG.debug("Unregistered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected SubtaskGateway getSubtaskGateway(int subtaskIndex) {
Review Comment:
`currentSubtaskGateway`, or `subtaskGateway` only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2821277080
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverCoordinator.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Coordinator for LockRemoverOperator. Handles lock release events from downstream operators. */
+@Experimental
Review Comment:
Remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822387420
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockRelease((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockTime = null;
+ }
+
+ @VisibleForTesting
+ void handleLockRelease(LockReleaseEvent event) {
+ Preconditions.checkArgument(lockTime != null, "Lock time is null, Can't release lock");
+
+ if (event.timestamp() >= lockTime) {
+ this.lockTime = null;
+ this.shouldRestoreTasks = false;
+ }
+ }
+
+ private void checkAndFire(ProcessingTimeService timerService) {
+ long current = timerService.getCurrentProcessingTime();
+ if (shouldRestoreTasks) {
+ if (lockTime != null) {
+ // Recovered tasks in progress. Skip trigger check
+ LOG.info("The recovery lock is still held at {}", current);
+ schedule(timerService, current + lockCheckDelayMs);
+ return;
+ }
+ }
+
+ Integer taskToStart =
+ TriggerUtil.nextTrigger(
+ evaluators, accumulatedChanges, lastTriggerTimes, current, startsFrom);
+ if (taskToStart == null) {
+ // Nothing to execute
+ if (!triggered) {
+ nothingToTriggerCounter.inc();
+ LOG.debug("Nothing to execute at {} for collected: {}", current, accumulatedChanges);
+ } else {
+ LOG.debug("Execution check finished");
+ }
+
+ // Next time start from the beginning
+ startsFrom = 0;
+ triggered = false;
+ return;
+ }
+
+ if (lockTime == null) {
+ this.lockTime = current;
+ TableChange change = accumulatedChanges.get(taskToStart);
+ output.collect(new StreamRecord<>(Trigger.create(current, taskToStart), current));
+ LOG.debug("Fired event with time: {}, collected: {} for {}", current, change, tableName);
+ triggerCounters.get(taskToStart).inc();
+ accumulatedChanges.set(taskToStart, TableChange.empty());
+ lastTriggerTimes.set(taskToStart, current);
+ schedule(timerService, current + minFireDelayMs);
+ startsFrom = (taskToStart + 1) % evaluators.size();
+ triggered = true;
+ } else {
+ // A task is already running, waiting for it to finish
+ LOG.info("Failed to acquire lock. Delaying task to {}", current + lockCheckDelayMs);
+
+ startsFrom = taskToStart;
+ concurrentRunThrottledCounter.inc();
+ schedule(timerService, current + lockCheckDelayMs);
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
Review Comment:
Remove it.
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockRelease((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
Review Comment:
Remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822410454
##########
flink/v2.1/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestLockRemoveCoordinator.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import org.apache.flink.runtime.operators.coordination.EventReceivingTasks;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.api.parallel.ExecutionMode;
+
+@Timeout(value = 10)
+@Execution(ExecutionMode.SAME_THREAD)
+class TestLockRemoveCoordinator extends TestBaseCoordinator {
+
+ private EventReceivingTasks receivingTasks;
+
+ @BeforeEach
+ public void before() {
+ this.receivingTasks = EventReceivingTasks.createForRunningTasks();
+ }
+
+ @Test
+ public void testEventHandling() throws Exception {
+ try (LockRemoverCoordinator lockRemoverCoordinator =
+ new LockRemoverCoordinator(
+ OPERATOR_NAME, new MockOperatorCoordinatorContext(TEST_OPERATOR_ID, 1)); ) {
Review Comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2822125652
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The TriggerManagerOperator itself holds the lock and registers a callback method with the
+ * coordinator. When a task finishes, it sends a signal from downstream to the coordinator to
+ * trigger this callback, allowing the TriggerManagerOperator to release the lock.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient Long lockTime;
+ private transient boolean shouldRestoreTasks = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ this.lockTime = current;
+ this.shouldRestoreTasks = true;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ this.lockTime = null;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleaseEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void onProcessingTime(long l) {
+ this.nextEvaluationTime = null;
+ checkAndFire(getProcessingTimeService());
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ this.lockTime = null;
+ }
+
+ @VisibleForTesting
+ void handleLockReleaseResult(LockReleaseEvent event) {
+ if (lockTime == null) {
+ return;
+ }
+
+ if (event.timestamp() >= lockTime) {
Review Comment:
Exactly, those methods are mutually exclusive. We don't need to synchronize between those methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2836107190
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
+
+ @SuppressWarnings("unchecked")
+ protected SubtaskGateways(String operatorName, int parallelism) {
+ this.operatorName = operatorName;
+ gateways = new Map[parallelism];
+
+ for (int i = 0; i < parallelism; ++i) {
+ gateways[i] = new java.util.HashMap<>();
+ }
+ }
+
+ protected void registerSubtaskGateway(SubtaskGateway gateway) {
+ int subtaskIndex = gateway.getSubtask();
+ int attemptNumber = gateway.getExecution().getAttemptNumber();
+ Preconditions.checkState(
+ !gateways[subtaskIndex].containsKey(attemptNumber),
+ "Coordinator of %s already has a subtask gateway for %d (#%d)",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ LOG.debug(
+ "Coordinator of {} registers gateway for subtask {} attempt {}",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ gateways[subtaskIndex].put(attemptNumber, gateway);
+ LOG.debug("Registered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected void unregisterSubtaskGateway(int subtaskIndex, int attemptNumber) {
+ gateways[subtaskIndex].remove(attemptNumber);
+ LOG.debug("Unregistered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected SubtaskGateway getSubtaskGateway(int subtaskIndex) {
Review Comment:
Change to subtaskGateway
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/BaseCoordinator.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base coordinator for table maintenance operators. Provides common functionality for thread
+ * management, subtask gateway management, and checkpoint handling.
+ */
+@Internal
+public abstract class BaseCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BaseCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private boolean started;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private final SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleaseEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private static final List<LockReleaseEvent> PENDING_RELEASE_EVENTS = Lists.newArrayList();
+
+ protected BaseCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "Coordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created coordinator: {}", operatorName);
+ }
+
+ void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @VisibleForTesting
+ void handleReleaseLock(LockReleaseEvent lockReleaseEvent) {
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (LOCK_RELEASE_CONSUMERS.containsKey(lockReleaseEvent.lockId())) {
+ LOCK_RELEASE_CONSUMERS.get(lockReleaseEvent.lockId()).accept(lockReleaseEvent);
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ } else {
+ PENDING_RELEASE_EVENTS.add(lockReleaseEvent);
+ LOG.info(
+ "No consumer for lock id {}, timestamp: {}",
+ lockReleaseEvent.lockId(),
+ lockReleaseEvent.timestamp());
+ }
+ }
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting coordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+
+ LOG.info("Closed coordinator: {}", operatorName);
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> resultFuture.complete(new byte[0]),
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
+ Preconditions.checkState(
+ !started, "The coordinator %s can only be reset if it was not yet started", operatorName);
+ LOG.info("Reset to checkpoint {}", checkpointId);
+ synchronized (PENDING_RELEASE_EVENTS) {
+ LOCK_RELEASE_CONSUMERS.clear();
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) {}
+
+ @Override
+ public void subtaskReset(int subtask, long checkpointId) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info("Subtask {} is reset to checkpoint {}", subtask, checkpointId);
+ Preconditions.checkState(coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.reset(subtask);
+ },
+ String.format(
+ Locale.ROOT, "handling subtask %d recovery to checkpoint %d", subtask, checkpointId));
+ }
+
+ @Override
+ public void executionAttemptFailed(int subtask, int attemptNumber, Throwable reason) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.info(
+ "Unregistering gateway after failure for subtask {} (#{}) of data statistics {}",
+ subtask,
+ attemptNumber,
+ operatorName);
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.unregisterSubtaskGateway(subtask, attemptNumber);
+ },
+ String.format(Locale.ROOT, "handling subtask %d (#%d) failure", subtask, attemptNumber));
+ }
+
+ @Override
+ public void executionAttemptReady(int subtask, int attemptNumber, SubtaskGateway gateway) {
+ Preconditions.checkArgument(subtask == gateway.getSubtask());
+ Preconditions.checkArgument(attemptNumber == gateway.getExecution().getAttemptNumber());
+ runInCoordinatorThread(
+ () -> {
+ Preconditions.checkState(
+ this.coordinatorThreadFactory.isCurrentThreadCoordinatorThread());
+ subtaskGateways.registerSubtaskGateway(gateway);
+ },
+ String.format(
+ Locale.ROOT,
+ "making event gateway to subtask %d (#%d) available",
+ subtask,
+ attemptNumber));
+ }
+
+ protected CoordinatorExecutorThreadFactory coordinatorThreadFactory() {
+ return coordinatorThreadFactory;
+ }
+
+ protected SubtaskGateways subtaskGateways() {
+ return subtaskGateways;
+ }
+
+ protected String operatorName() {
+ return operatorName;
+ }
+
+ protected void runInCoordinatorThread(Runnable runnable, String actionString) {
+ ensureStarted();
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ runnable.run();
+ } catch (Throwable t) {
+ LOG.error(
+ "Uncaught exception in coordinator while {}: {}", actionString, t.getMessage(), t);
+ context.failJob(t);
+ }
+ });
+ }
+
+ @VisibleForTesting
+ void callInCoordinatorThread(Callable<Void> callable, String errorMessage) {
+ ensureStarted();
+ // Ensure the task is done by the coordinator executor.
+ if (!coordinatorThreadFactory.isCurrentThreadCoordinatorThread()) {
+ try {
+ Callable<Void> guardedCallable =
+ () -> {
+ try {
+ return callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ ExceptionUtils.rethrowException(t);
+ return null;
+ }
+ };
+
+ coordinatorExecutor.submit(guardedCallable).get();
+ } catch (InterruptedException | ExecutionException e) {
+ throw new FlinkRuntimeException(errorMessage, e);
+ }
+ } else {
+ try {
+ callable.call();
+ } catch (Throwable t) {
+ LOG.error("Uncaught Exception in coordinator {} executor", operatorName, t);
+ throw new FlinkRuntimeException(errorMessage, t);
+ }
+ }
+ }
+
+ private void ensureStarted() {
+ Preconditions.checkState(started, "The coordinator has not started yet.");
+ }
+
+ /** Inner class to manage subtask gateways. */
+ protected static class SubtaskGateways {
+ private final String operatorName;
+ private final Map<Integer, SubtaskGateway>[] gateways;
+
+ @SuppressWarnings("unchecked")
+ protected SubtaskGateways(String operatorName, int parallelism) {
+ this.operatorName = operatorName;
+ gateways = new Map[parallelism];
+
+ for (int i = 0; i < parallelism; ++i) {
+ gateways[i] = new java.util.HashMap<>();
+ }
+ }
+
+ protected void registerSubtaskGateway(SubtaskGateway gateway) {
+ int subtaskIndex = gateway.getSubtask();
+ int attemptNumber = gateway.getExecution().getAttemptNumber();
+ Preconditions.checkState(
+ !gateways[subtaskIndex].containsKey(attemptNumber),
+ "Coordinator of %s already has a subtask gateway for %d (#%d)",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ LOG.debug(
+ "Coordinator of {} registers gateway for subtask {} attempt {}",
+ operatorName,
+ subtaskIndex,
+ attemptNumber);
+ gateways[subtaskIndex].put(attemptNumber, gateway);
+ LOG.debug("Registered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected void unregisterSubtaskGateway(int subtaskIndex, int attemptNumber) {
+ gateways[subtaskIndex].remove(attemptNumber);
+ LOG.debug("Unregistered gateway for subtask {} attempt {}", subtaskIndex, attemptNumber);
+ }
+
+ protected SubtaskGateway getSubtaskGateway(int subtaskIndex) {
+ Preconditions.checkState(
+ !gateways[subtaskIndex].isEmpty(),
+ "Coordinator subtask %d is not ready yet to receive events",
+ subtaskIndex);
+ return gateways[subtaskIndex].values().iterator().next();
+ }
+
+ protected void reset(int subtaskIndex) {
+ gateways[subtaskIndex].clear();
+ }
+ }
+
+ /** Custom thread factory for the coordinator executor. */
+ protected static class CoordinatorExecutorThreadFactory
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816178502
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerCoordinator.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TriggerManagerCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerCoordinator.class);
+
+ public TriggerManagerCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created TriggerManagerCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerLock((LockRegisterEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerLock(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
+ lockRegisterEvent.lockId(),
+ lock -> {
+ LOG.info(
+ "Send release event for lock id {}, timestamp: {} to Operator {}",
+ lock.lockId(),
+ lock.timestamp(),
+ operatorName());
+ subtaskGateways().getSubtaskGateway(0).sendEvent(lock);
+ });
+
+ synchronized (PENDING_RELEASE_EVENTS) {
+ if (!PENDING_RELEASE_EVENTS.isEmpty()) {
+ PENDING_RELEASE_EVENTS.forEach(this::handleReleaseLock);
+ PENDING_RELEASE_EVENTS.clear();
+ }
+ }
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // We don’t need to track how many locks are currently held, because when recovering from state,
+ // a `recover lock` will be issued to ensure all tasks finish running and then release all
+ // locks.
+ // The `TriggerManagerOperator` already keeps the `TableChange` state and related information,
+ // so there’s no need to store additional state here.
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, byte[] checkpointData) {
Review Comment:
Are we sure that the calls to the `resetToCheckpoint` are not intermixed with the actual message handling?
Consider:
1. Job restarts
2. LockRemoverCoordinator, LockRemoveOperator starts
3. TaskManagerCoordinator, TaskManagerOperator starts
4. TaskManagerOperator sends the register event
5. TaskManagerOperator sends the Trigger
6. LockRemoveOperator sends the LockReleaseEvent
7. LockManagerCoordinator stores the event in the `PENDING_RELEASE_EVENTS`
8. TaskManagerCoordinator calls the `resetToCheckpoint`
9. Deadlock
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2816113760
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverCoordinator.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.Locale;
+import java.util.concurrent.CompletableFuture;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Coordinator for LockRemoverOperator. Handles lock release events from downstream operators. */
+@Experimental
+@Internal
+public class LockRemoverCoordinator extends BaseCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(LockRemoverCoordinator.class);
+
+ public LockRemoverCoordinator(String operatorName, Context context) {
+ super(operatorName, context);
+ LOG.info("Created LockRemoverCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName(),
+ event);
+ if (event instanceof LockReleaseEvent) {
+ handleReleaseLock((LockReleaseEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @Override
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
+ throws Exception {
+ runInCoordinatorThread(
+ () -> {
+ resultFuture.complete(new byte[0]);
+ },
+ String.format(Locale.ROOT, "taking checkpoint %d", checkpointId));
+ }
+
+ @Override
+ public void resetToCheckpoint(long checkpointId, @Nullable byte[] checkpointData)
+ throws Exception {}
Review Comment:
Maybe move this to the base too? It is just boilerplate and overridden in the LMCoordinator anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#issuecomment-3817401515
@mxm This PR is an improvement based on the comments in #15042, and I hope it will replace https://github.com/apache/iceberg/pull/15042. However, I’m worried that #15042 might still be in use, so I haven’t closed it for now. Please treat this PR as the primary one, and I apologize for any confusion this may cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2736145487
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TableMaintenanceCoordinator.java:
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.function.Consumer;
+import javax.annotation.Nonnull;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FatalExitExceptionHandler;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class TableMaintenanceCoordinator implements OperatorCoordinator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableMaintenanceCoordinator.class);
+
+ private final String operatorName;
+ private final Context context;
+
+ private final ExecutorService coordinatorExecutor;
+ private final CoordinatorExecutorThreadFactory coordinatorThreadFactory;
+ private transient boolean started;
+ private final transient SubtaskGateways subtaskGateways;
+ private static final Map<String, Consumer<LockReleasedEvent>> LOCK_RELEASE_CONSUMERS =
+ Maps.newConcurrentMap();
+ private transient List<LockReleasedEvent> pendingReleaseEvents = Lists.newArrayList();
+
+ public TableMaintenanceCoordinator(String operatorName, Context context) {
+ this.operatorName = operatorName;
+ this.context = context;
+
+ this.coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "TableMaintenanceCoordinator-" + operatorName, context.getUserCodeClassloader());
+ this.coordinatorExecutor = Executors.newSingleThreadExecutor(coordinatorThreadFactory);
+ this.subtaskGateways = new SubtaskGateways(operatorName, context.currentParallelism());
+ LOG.info("Created TableMaintenanceCoordinator: {}", operatorName);
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting TableMaintenanceCoordinator: {}", operatorName);
+ this.started = true;
+ }
+
+ @Override
+ public void close() throws Exception {
+ coordinatorExecutor.shutdown();
+ this.started = false;
+ LOG.info("Closed TableMaintenanceCoordinator: {}", operatorName);
+ LOCK_RELEASE_CONSUMERS.clear();
+ pendingReleaseEvents.clear();
+ }
+
+ @Override
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInCoordinatorThread(
+ () -> {
+ LOG.debug(
+ "Handling event from subtask {} (#{}) of {}: {}",
+ subtask,
+ attemptNumber,
+ operatorName,
+ event);
+ if (event instanceof LockRegisterEvent) {
+ registerTriggerManagerReceiveReleaseEvent((LockRegisterEvent) event);
+ } else if (event instanceof LockReleasedEvent) {
+ handleReleaseLock((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ },
+ String.format(
+ Locale.ROOT,
+ "handling operator event %s from subtask %d (#%d)",
+ event.getClass(),
+ subtask,
+ attemptNumber));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void registerTriggerManagerReceiveReleaseEvent(LockRegisterEvent lockRegisterEvent) {
+ LOCK_RELEASE_CONSUMERS.put(
Review Comment:
TriggerManagerOperator and LockRemoverOperator both are forceNonParallel, so they will only 1 subtask, so for now I haven’t taken registrations from different subtasks into account.
If the job restarts, it will re-register during `initializeState`, overwriting the previous one so that we always have the latest reference.So is it still necessary to do any checks here, and if so, what exactly should we be checking for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2736376231
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/LockRemoverOperator.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.iceberg.flink.maintenance.api.TaskResult;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Experimental
+@Internal
+public class LockRemoverOperator extends AbstractStreamOperator<Void>
+ implements OneInputStreamOperator<TaskResult, Void>, OperatorEventHandler {
+ private static final Logger LOG = LoggerFactory.getLogger(LockRemoverOperator.class);
+
+ private static final long serialVersionUID = 1L;
+
+ private final String tableName;
+ private final OperatorEventGateway operatorEventGateway;
+ private final List<String> maintenanceTaskNames;
+ private transient List<Counter> succeededTaskResultCounters;
+ private transient List<Counter> failedTaskResultCounters;
+ private transient List<AtomicLong> taskLastRunDurationMs;
+
+ LockRemoverOperator(
+ StreamOperatorParameters<Void> parameters,
+ OperatorEventGateway operatorEventGateway,
+ String tableName,
+ List<String> maintenanceTaskNames) {
+ super(parameters);
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ }
+
+ @Override
+ public void open() throws Exception {
+ this.succeededTaskResultCounters =
+ Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.failedTaskResultCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ this.taskLastRunDurationMs = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ MetricGroup taskMetricGroup =
+ TableMaintenanceMetrics.groupFor(
+ getRuntimeContext(), tableName, maintenanceTaskNames.get(taskIndex), taskIndex);
+ succeededTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.SUCCEEDED_TASK_COUNTER));
+ failedTaskResultCounters.add(
+ taskMetricGroup.counter(TableMaintenanceMetrics.FAILED_TASK_COUNTER));
+ AtomicLong duration = new AtomicLong(0);
+ taskLastRunDurationMs.add(duration);
+ taskMetricGroup.gauge(TableMaintenanceMetrics.LAST_RUN_DURATION_MS, duration::get);
+ }
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ // no incoming events
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TaskResult> streamRecord) {
+ TaskResult taskResult = streamRecord.getValue();
+ LOG.info(
+ "Processing result {} for task {}",
+ taskResult,
+ maintenanceTaskNames.get(taskResult.taskIndex()));
+ long duration = System.currentTimeMillis() - taskResult.startEpoch();
+ operatorEventGateway.sendEventToCoordinator(
+ new LockReleasedEvent(tableName, streamRecord.getTimestamp()));
Review Comment:
I change this part to use the “lockTime” in a Long instead of two boolean flag ,and use keep only one place to send release event.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: TableMaintenance Support Coordinator Lock [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15151:
URL: https://github.com/apache/iceberg/pull/15151#discussion_r2735410342
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManagerOperator.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink.maintenance.operator;
+
+import java.util.List;
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEventGateway;
+import org.apache.flink.runtime.operators.coordination.OperatorEventHandler;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.iceberg.flink.maintenance.api.Trigger;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TriggerManagerOperator sends events to the coordinator to acquire a lock, then waits for the
+ * response. If the response indicates that the lock has been acquired, it fires a trigger;
+ * otherwise, it schedules the next attempt. When the job recovers from a failure, tasks from
+ * different execution paths of the previous run may still be running. Therefore, it first needs to
+ * send a lock with the maximum timestamp, and then send a recovery trigger. Only after the
+ * downstream removes this lock can we be sure that all tasks have fully stopped.
+ */
+@Experimental
+@Internal
+public class TriggerManagerOperator extends AbstractStreamOperator<Trigger>
+ implements OneInputStreamOperator<TableChange, Trigger>,
+ OperatorEventHandler,
+ ProcessingTimeCallback {
+ private static final Logger LOG = LoggerFactory.getLogger(TriggerManagerOperator.class);
+
+ private final List<String> maintenanceTaskNames;
+ private final List<TriggerEvaluator> evaluators;
+ private transient Long nextEvaluationTime;
+ private final long minFireDelayMs;
+ private final OperatorEventGateway operatorEventGateway;
+ private transient List<TableChange> accumulatedChanges;
+ private transient ListState<Long> nextEvaluationTimeState;
+ private transient ListState<TableChange> accumulatedChangesState;
+ private transient ListState<Long> lastTriggerTimesState;
+ private transient Counter rateLimiterTriggeredCounter;
+ private transient Counter concurrentRunThrottledCounter;
+ private transient Counter nothingToTriggerCounter;
+ private transient List<Counter> triggerCounters;
+ private final long lockCheckDelayMs;
+ private transient List<Long> lastTriggerTimes;
+ // To keep the task scheduling fair we keep the last triggered task position in memory.
+ // If we find a task to trigger, then we run it, but after it is finished, we start from the given
+ // position to prevent "starvation" of the tasks.
+ // When there is nothing to trigger, we start from the beginning, as the order of the tasks might
+ // be important (RewriteDataFiles first, and then RewriteManifestFiles later)
+ private transient int startsFrom = 0;
+ private transient boolean triggered = false;
+ private final String tableName;
+ private transient boolean restoreTasks = false;
+ private transient boolean lockHeld = false;
+
+ public TriggerManagerOperator(
+ StreamOperatorParameters<Trigger> parameters,
+ OperatorEventGateway operatorEventGateway,
+ List<String> maintenanceTaskNames,
+ List<TriggerEvaluator> evaluators,
+ long minFireDelayMs,
+ long lockCheckDelayMs,
+ String tableName) {
+ super(parameters);
+ Preconditions.checkArgument(
+ maintenanceTaskNames != null && !maintenanceTaskNames.isEmpty(),
+ "Invalid maintenance task names: null or empty");
+ Preconditions.checkArgument(
+ evaluators != null && !evaluators.isEmpty(), "Invalid evaluators: null or empty");
+ Preconditions.checkArgument(
+ maintenanceTaskNames.size() == evaluators.size(),
+ "Provide a name and evaluator for all of the maintenance tasks");
+ Preconditions.checkArgument(minFireDelayMs > 0, "Minimum fire delay should be at least 1.");
+ Preconditions.checkArgument(
+ lockCheckDelayMs > 0, "Minimum lock delay rate should be at least 1 ms.");
+
+ this.maintenanceTaskNames = maintenanceTaskNames;
+ this.evaluators = evaluators;
+ this.minFireDelayMs = minFireDelayMs;
+ this.lockCheckDelayMs = lockCheckDelayMs;
+ this.tableName = tableName;
+ this.operatorEventGateway = operatorEventGateway;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ MetricGroup mainGroup = TableMaintenanceMetrics.groupFor(getRuntimeContext(), tableName);
+ this.rateLimiterTriggeredCounter =
+ mainGroup.counter(TableMaintenanceMetrics.RATE_LIMITER_TRIGGERED);
+ this.concurrentRunThrottledCounter =
+ mainGroup.counter(TableMaintenanceMetrics.CONCURRENT_RUN_THROTTLED);
+ this.nothingToTriggerCounter = mainGroup.counter(TableMaintenanceMetrics.NOTHING_TO_TRIGGER);
+ this.triggerCounters = Lists.newArrayListWithExpectedSize(maintenanceTaskNames.size());
+ for (int taskIndex = 0; taskIndex < maintenanceTaskNames.size(); ++taskIndex) {
+ triggerCounters.add(
+ TableMaintenanceMetrics.groupFor(
+ mainGroup, maintenanceTaskNames.get(taskIndex), taskIndex)
+ .counter(TableMaintenanceMetrics.TRIGGERED));
+ }
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+ this.nextEvaluationTimeState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerNextTriggerTime", Types.LONG));
+
+ this.accumulatedChangesState =
+ context
+ .getOperatorStateStore()
+ .getListState(
+ new ListStateDescriptor<>(
+ "triggerManagerAccumulatedChange", TypeInformation.of(TableChange.class)));
+
+ this.lastTriggerTimesState =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("triggerManagerLastTriggerTime", Types.LONG));
+
+ long current = getProcessingTimeService().getCurrentProcessingTime();
+
+ // Initialize from state
+ if (!Iterables.isEmpty(nextEvaluationTimeState.get())) {
+ nextEvaluationTime = Iterables.getOnlyElement(nextEvaluationTimeState.get());
+ }
+
+ this.accumulatedChanges = Lists.newArrayList(accumulatedChangesState.get());
+ this.lastTriggerTimes = Lists.newArrayList(lastTriggerTimesState.get());
+
+ // Initialize if the state was empty
+ if (accumulatedChanges.isEmpty()) {
+ for (int i = 0; i < evaluators.size(); ++i) {
+ accumulatedChanges.add(TableChange.empty());
+ lastTriggerTimes.add(current);
+ }
+ }
+
+ // register the lock register event
+ operatorEventGateway.sendEventToCoordinator(new LockRegisterEvent(tableName, current));
+
+ if (context.isRestored()) {
+ // When the job state is restored, there could be ongoing tasks.
+ // To prevent collision with the new triggers the following is done:
+ // - add a recovery lock
+ // - fire a recovery trigger
+ // This ensures that the tasks of the previous trigger are executed, and the lock is removed
+ // in the end. The result of the 'tryLock' is ignored as an already existing lock prevents
+ // collisions as well.
+ // register the recover lock
+ restoreTasks = true;
+ lockHeld = false;
+ output.collect(new StreamRecord<>(Trigger.recovery(current), current));
+ if (nextEvaluationTime == null) {
+ schedule(getProcessingTimeService(), current + minFireDelayMs);
+ } else {
+ schedule(getProcessingTimeService(), nextEvaluationTime);
+ }
+ } else {
+ restoreTasks = false;
+ lockHeld = false;
+ }
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ nextEvaluationTimeState.clear();
+ if (nextEvaluationTime != null) {
+ nextEvaluationTimeState.add(nextEvaluationTime);
+ }
+
+ accumulatedChangesState.update(accumulatedChanges);
+ lastTriggerTimesState.update(lastTriggerTimes);
+ LOG.info(
+ "Storing state: nextEvaluationTime {}, accumulatedChanges {}, lastTriggerTimes {}",
+ nextEvaluationTime,
+ accumulatedChanges,
+ lastTriggerTimes);
+ }
+
+ @Override
+ public void handleOperatorEvent(OperatorEvent event) {
+ if (event instanceof LockReleasedEvent) {
+ LOG.info("Received lock released event: {}", event);
+ handleLockReleaseResult((LockReleasedEvent) event);
+ } else {
+ throw new IllegalArgumentException(
+ "Invalid operator event type: " + event.getClass().getCanonicalName());
+ }
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ @Override
+ public void processElement(StreamRecord<TableChange> streamRecord) throws Exception {
+ TableChange change = streamRecord.getValue();
+ accumulatedChanges.forEach(tableChange -> tableChange.merge(change));
+ if (nextEvaluationTime == null) {
+ checkAndFire(getProcessingTimeService());
+ } else {
+ LOG.info(
+ "Trigger manager rate limiter triggered current: {}, next: {}, accumulated changes: {},{}",
+ getProcessingTimeService().getCurrentProcessingTime(),
+ nextEvaluationTime,
+ accumulatedChanges,
+ maintenanceTaskNames);
+ rateLimiterTriggeredCounter.inc();
+ }
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ super.processWatermark(mark);
+ }
Review Comment:
This is not needed 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org