You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by do...@apache.org on 2026/06/01 17:15:47 UTC
(seatunnel) branch dev updated: [Feature][Core] Support table-level fault isolation for multi-table synchronization (#10600)
This is an automated email from the ASF dual-hosted git repository.
dybyte pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 2739847410 [Feature][Core] Support table-level fault isolation for multi-table synchronization (#10600)
2739847410 is described below
commit 27398474108c74523cd49e634eba7a0b0eaa6752
Author: yzeng1618 <yz...@gmail.com>
AuthorDate: Tue Jun 2 01:15:38 2026 +0800
[Feature][Core] Support table-level fault isolation for multi-table synchronization (#10600)
Co-authored-by: zengyi <ze...@chinatelecom.cn>
---
docs/en/architecture/features/multi-table.md | 19 +
docs/zh/architecture/features/multi-table.md | 21 +
.../common/multitable/MultiTableFailedTable.java | 55 ++
.../common/multitable/MultiTableFailureHelper.java | 250 ++++++++++
.../common/multitable/MultiTableFailurePhase.java | 40 ++
.../seatunnel/api/options/EnvOptionRule.java | 1 +
.../api/options/MultiTableCommonOptions.java | 46 ++
.../api/options/MultiTableFailurePolicy.java | 28 ++
.../api/sink/multitablesink/MultiTableSink.java | 70 ++-
.../sink/multitablesink/MultiTableSinkWriter.java | 474 +++++++++++++++++-
.../multitablesink/MultiTableWriterRunnable.java | 178 ++++++-
.../seatunnel/api/table/catalog/Catalog.java | 57 ++-
.../multitablesink/MultiTableSinkWriterTest.java | 551 +++++++++++++++++++++
.../api/table/catalog/CatalogTableTest.java | 35 ++
.../jdbc/catalog/duckdb/DuckDBCatalog.java | 4 +-
.../seatunnel/jdbc/config/JdbcSourceConfig.java | 5 +
.../seatunnel/jdbc/source/JdbcSource.java | 3 +-
.../seatunnel/jdbc/utils/JdbcCatalogUtils.java | 103 +++-
.../flink/execution/SinkExecuteProcessor.java | 118 ++++-
.../execution/AbstractSinkExecuteProcessor.java | 119 ++++-
.../spark/execution/SinkExecuteProcessor.java | 129 ++++-
.../spark/execution/SinkExecuteProcessor.java | 126 ++++-
.../connectors/seatunnel/jdbc/AbstractJdbcIT.java | 4 +-
.../seatunnel/jdbc/JdbcMysqlMultipleTablesIT.java | 40 ++
...h_missing_sink_table_continue_other_tables.conf | 69 +++
.../client/MultipleTableJobConfigParserTest.java | 215 ++++++++
.../core/parse/MultipleTableJobConfigParser.java | 199 ++++++--
.../engine/server/dag/physical/SubPlan.java | 5 +-
.../seatunnel/engine/server/master/JobMaster.java | 48 +-
29 files changed, 2815 insertions(+), 197 deletions(-)
diff --git a/docs/en/architecture/features/multi-table.md b/docs/en/architecture/features/multi-table.md
index 477e16bec0..ff25365feb 100644
--- a/docs/en/architecture/features/multi-table.md
+++ b/docs/en/architecture/features/multi-table.md
@@ -683,6 +683,25 @@ sink {
Error tolerance and retry policies are typically connector-specific. Avoid relying on undocumented `multi-table.*` option keys unless they are defined by the connector you use.
+SeaTunnel also provides a framework-level failure policy for multi-table jobs:
+
+```hocon
+env {
+ multi_table {
+ failure_policy = "CONTINUE_OTHER_TABLES"
+ }
+}
+```
+
+When `failure_policy` is set to `CONTINUE_OTHER_TABLES`:
+
+- table-scoped failures during table discovery, sink initialization, save mode handling, or `MultiTableSink` runtime writes are recorded and printed with `table`, `phase`, `plugin`, `exception`, and summarized `reason`
+- healthy tables continue to run instead of being blocked by a small number of abnormal tables
+- batch jobs still finish as `FAILED` if any table was isolated
+- streaming jobs keep running while healthy tables remain active
+
+Shared failures such as source connection loss, checkpoint coordinator failures, plugin loading failures, or OOM conditions still abort the whole job.
+
## 11. Limitations and Considerations
### 11.1 Current Limitations
diff --git a/docs/zh/architecture/features/multi-table.md b/docs/zh/architecture/features/multi-table.md
index 966135640e..611656226c 100644
--- a/docs/zh/architecture/features/multi-table.md
+++ b/docs/zh/architecture/features/multi-table.md
@@ -422,6 +422,27 @@ sink {
}
```
+### 10.4 异常表隔离
+
+对于多表任务,可以在 `env` 中开启框架级异常表隔离:
+
+```hocon
+env {
+ multi_table {
+ failure_policy = "CONTINUE_OTHER_TABLES"
+ }
+}
+```
+
+启用后:
+
+- 启动阶段的表发现、sink 初始化、save mode 处理,以及 `MultiTableSink` 运行期写入失败,会按表打印 `table`、`phase`、`plugin`、`exception`、`reason`
+- 健康表继续运行,异常表被隔离,不再拖垮其余表
+- Batch 作业只要出现失败表,最终状态仍为 `FAILED`
+- Streaming 作业只要还有健康表,会继续保持运行
+
+该配置不处理共享故障,例如 source 连接中断、checkpoint 协调失败、插件加载失败或 OOM;这类异常仍会导致整个作业失败。
+
## 13. 相关资源
- [CatalogTable 和元数据](../api-design/catalog-table.md)
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/common/multitable/MultiTableFailedTable.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/common/multitable/MultiTableFailedTable.java
new file mode 100644
index 0000000000..c296106b3d
--- /dev/null
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/common/multitable/MultiTableFailedTable.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.api.common.multitable;
+
+import lombok.Getter;
+
+import java.io.Serializable;
+
+/** A normalized failed-table record used by parser and runtime multi-table components. */
+@Getter
+public class MultiTableFailedTable implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final String tablePath;
+ private final MultiTableFailurePhase phase;
+ private final String pluginName;
+ private final String exceptionClass;
+ private final String messageSummary;
+ private final long firstFailureTime;
+ // Throwable chains may capture connector internals that are not safe to ship with job graph
+ // serialization. Failed-table metadata is the stable contract across stages.
+ private final transient Throwable cause;
+
+ public MultiTableFailedTable(
+ String tablePath,
+ MultiTableFailurePhase phase,
+ String pluginName,
+ String exceptionClass,
+ String messageSummary,
+ long firstFailureTime,
+ Throwable cause) {
+ this.tablePath = tablePath;
+ this.phase = phase;
+ this.pluginName = pluginName;
+ this.exceptionClass = exceptionClass;
+ this.messageSummary = messageSummary;
+ this.firstFailureTime = firstFailureTime;
+ this.cause = cause;
+ }
+}
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/common/multitable/MultiTableFailureHelper.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/common/multitable/MultiTableFailureHelper.java
new file mode 100644
index 0000000000..ca87980ef0
--- /dev/null
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/common/multitable/MultiTableFailureHelper.java
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.api.common.multitable;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.options.MultiTableCommonOptions;
+import org.apache.seatunnel.api.options.MultiTableFailurePolicy;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public final class MultiTableFailureHelper {
+
+ public static final String ISOLATED_FAILURE_MARKER = "[MULTI_TABLE_ISOLATED_FAILURE]";
+
+ private static final String FAILED_TABLE_DELIMITER = ":";
+ private static final int FAILED_TABLE_FIELD_COUNT = 6;
+ private static final ThreadLocal<List<MultiTableFailedTable>> FAILED_TABLE_COLLECTOR =
+ new ThreadLocal<>();
+
+ private MultiTableFailureHelper() {}
+
+ public static boolean shouldContinueOtherTables(ReadonlyConfig options) {
+ return options != null
+ && options.get(MultiTableCommonOptions.MULTI_TABLE_FAILURE_POLICY)
+ .continueOtherTables();
+ }
+
+ public static ReadonlyConfig mergeOptions(ReadonlyConfig primary, ReadonlyConfig fallback) {
+ if (primary == null) {
+ return fallback;
+ }
+ if (fallback == null) {
+ return primary;
+ }
+ return ReadonlyConfig.fromConfig(primary.toConfig().withFallback(fallback.toConfig()));
+ }
+
+ public static ReadonlyConfig withMultiTableFailurePolicy(
+ ReadonlyConfig options, ReadonlyConfig envOptions) {
+ if (envOptions == null) {
+ return options;
+ }
+ Map<String, Object> internalOptions = new HashMap<>();
+ MultiTableFailurePolicy failurePolicy =
+ envOptions.get(MultiTableCommonOptions.MULTI_TABLE_FAILURE_POLICY);
+ internalOptions.put(
+ MultiTableCommonOptions.MULTI_TABLE_FAILURE_POLICY.key(), failurePolicy.name());
+ return mergeOptions(ReadonlyConfig.fromMap(internalOptions), options);
+ }
+
+ public static ReadonlyConfig withFailedTables(
+ ReadonlyConfig options, Collection<MultiTableFailedTable> failedTables) {
+ if (failedTables == null || failedTables.isEmpty()) {
+ return options;
+ }
+ Map<String, Object> internalOptions = new HashMap<>();
+ internalOptions.put(
+ MultiTableCommonOptions.MULTI_TABLE_INITIAL_FAILED_TABLES.key(),
+ serializeFailedTables(failedTables));
+ return mergeOptions(ReadonlyConfig.fromMap(internalOptions), options);
+ }
+
+ public static MultiTableFailedTable buildFailedTable(
+ String tablePath, MultiTableFailurePhase phase, String pluginName, Throwable error) {
+ String message = summarizeThrowable(error);
+ return new MultiTableFailedTable(
+ tablePath,
+ phase,
+ pluginName,
+ error == null ? "UnknownException" : error.getClass().getSimpleName(),
+ message,
+ System.currentTimeMillis(),
+ error);
+ }
+
+ public static <T> T collectFailedTables(
+ Collection<MultiTableFailedTable> target, Supplier<T> action) {
+ List<MultiTableFailedTable> previous = FAILED_TABLE_COLLECTOR.get();
+ List<MultiTableFailedTable> current = new ArrayList<>();
+ FAILED_TABLE_COLLECTOR.set(current);
+ try {
+ return action.get();
+ } finally {
+ if (target != null) {
+ target.addAll(current);
+ }
+ if (previous == null) {
+ FAILED_TABLE_COLLECTOR.remove();
+ } else {
+ previous.addAll(current);
+ FAILED_TABLE_COLLECTOR.set(previous);
+ }
+ }
+ }
+
+ public static void recordFailedTable(MultiTableFailedTable failedTable) {
+ List<MultiTableFailedTable> failedTables = FAILED_TABLE_COLLECTOR.get();
+ if (failedTables != null && failedTable != null) {
+ failedTables.add(failedTable);
+ }
+ }
+
+ public static List<MultiTableFailedTable> getInitialFailedTables(ReadonlyConfig options) {
+ if (options == null) {
+ return Collections.emptyList();
+ }
+ return options.getOptional(MultiTableCommonOptions.MULTI_TABLE_INITIAL_FAILED_TABLES)
+ .orElse(Collections.emptyList()).stream()
+ .map(MultiTableFailureHelper::deserializeFailedTable)
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList());
+ }
+
+ public static String formatFailedTableSummary(
+ String title, Collection<MultiTableFailedTable> failedTables) {
+ if (failedTables == null || failedTables.isEmpty()) {
+ return title;
+ }
+ Map<String, MultiTableFailedTable> sortedFailures = new TreeMap<>();
+ failedTables.forEach(failure -> sortedFailures.put(failure.getTablePath(), failure));
+ return title
+ + System.lineSeparator()
+ + sortedFailures.values().stream()
+ .map(MultiTableFailureHelper::formatFailedTableLine)
+ .collect(Collectors.joining(System.lineSeparator()));
+ }
+
+ public static String withIsolatedFailureMarker(String message) {
+ if (message == null || message.isEmpty()) {
+ return ISOLATED_FAILURE_MARKER;
+ }
+ if (isIsolatedFailure(message)) {
+ return message;
+ }
+ return ISOLATED_FAILURE_MARKER + System.lineSeparator() + message;
+ }
+
+ public static boolean isIsolatedFailure(String message) {
+ return message != null && message.contains(ISOLATED_FAILURE_MARKER);
+ }
+
+ public static String formatFailedTableLine(MultiTableFailedTable failure) {
+ return String.format(
+ "table=%s, phase=%s, plugin=%s, exception=%s, reason=%s",
+ failure.getTablePath(),
+ failure.getPhase(),
+ failure.getPluginName(),
+ failure.getExceptionClass(),
+ failure.getMessageSummary());
+ }
+
+ public static String summarizeThrowable(Throwable error) {
+ Throwable current = error;
+ String message = null;
+ while (current != null) {
+ if (current.getMessage() != null && !current.getMessage().trim().isEmpty()) {
+ message = current.getMessage().trim();
+ }
+ current = current.getCause();
+ }
+ if (message == null || message.isEmpty()) {
+ message = error == null ? "" : error.getClass().getName();
+ }
+ message = message.replaceAll("\\s+", " ");
+ if (message.length() > 300) {
+ return message.substring(0, 297) + "...";
+ }
+ return message;
+ }
+
+ private static List<String> serializeFailedTables(
+ Collection<MultiTableFailedTable> failedTables) {
+ return failedTables.stream()
+ .map(MultiTableFailureHelper::serializeFailedTable)
+ .collect(Collectors.toList());
+ }
+
+ private static String serializeFailedTable(MultiTableFailedTable failedTable) {
+ return Stream.of(
+ failedTable.getTablePath(),
+ failedTable.getPhase().name(),
+ failedTable.getPluginName(),
+ failedTable.getExceptionClass(),
+ failedTable.getMessageSummary(),
+ String.valueOf(failedTable.getFirstFailureTime()))
+ .map(MultiTableFailureHelper::encodeField)
+ .collect(Collectors.joining(FAILED_TABLE_DELIMITER));
+ }
+
+ private static MultiTableFailedTable deserializeFailedTable(String serialized) {
+ try {
+ String[] parts = serialized.split(FAILED_TABLE_DELIMITER, -1);
+ if (parts.length != FAILED_TABLE_FIELD_COUNT) {
+ return null;
+ }
+ return new MultiTableFailedTable(
+ decodeField(parts[0]),
+ MultiTableFailurePhase.valueOf(decodeField(parts[1])),
+ decodeField(parts[2]),
+ decodeField(parts[3]),
+ decodeField(parts[4]),
+ Long.parseLong(decodeField(parts[5])),
+ null);
+ } catch (RuntimeException ignore) {
+ return null;
+ }
+ }
+
+ private static String encodeField(String value) {
+ return Base64.getUrlEncoder()
+ .encodeToString((value == null ? "" : value).getBytes(StandardCharsets.UTF_8));
+ }
+
+ private static String decodeField(String value) {
+ return new String(Base64.getUrlDecoder().decode(value), StandardCharsets.UTF_8);
+ }
+
+ public static boolean isFailFast(ReadonlyConfig options) {
+ return options == null
+ || options.get(MultiTableCommonOptions.MULTI_TABLE_FAILURE_POLICY)
+ == MultiTableFailurePolicy.FAIL_FAST;
+ }
+}
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/common/multitable/MultiTableFailurePhase.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/common/multitable/MultiTableFailurePhase.java
new file mode 100644
index 0000000000..3fafecef94
--- /dev/null
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/common/multitable/MultiTableFailurePhase.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.api.common.multitable;
+
+/** Lifecycle phase where a multi-table failure happened. */
+public enum MultiTableFailurePhase {
+ DISCOVERY("discovery"),
+ SOURCE_INIT("source_init"),
+ SINK_INIT("sink_init"),
+ SAVE_MODE("save_mode"),
+ RUNTIME_WRITE("runtime_write"),
+ CHECKPOINT("checkpoint"),
+ CLOSE("close");
+
+ private final String phase;
+
+ MultiTableFailurePhase(String phase) {
+ this.phase = phase;
+ }
+
+ @Override
+ public String toString() {
+ return phase;
+ }
+}
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/EnvOptionRule.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/EnvOptionRule.java
index bb025aa4d2..cce5229f4c 100644
--- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/EnvOptionRule.java
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/EnvOptionRule.java
@@ -46,6 +46,7 @@ public class EnvOptionRule implements Factory {
EnvCommonOptions.READ_LIMIT_ROW_PER_SECOND,
EnvCommonOptions.READ_LIMIT_BYTES_PER_SECOND,
EnvCommonOptions.SAVEMODE_EXECUTE_LOCATION,
+ MultiTableCommonOptions.MULTI_TABLE_FAILURE_POLICY,
EnvCommonOptions.CUSTOM_PARAMETERS,
EnvCommonOptions.NODE_TAG_FILTER)
.build();
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/MultiTableCommonOptions.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/MultiTableCommonOptions.java
new file mode 100644
index 0000000000..3723f98022
--- /dev/null
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/MultiTableCommonOptions.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.api.options;
+
+import org.apache.seatunnel.api.annotation.Experimental;
+import org.apache.seatunnel.api.configuration.Option;
+import org.apache.seatunnel.api.configuration.Options;
+
+import java.util.List;
+
+public class MultiTableCommonOptions {
+
+ @Experimental
+ public static final Option<MultiTableFailurePolicy> MULTI_TABLE_FAILURE_POLICY =
+ Options.key("multi_table.failure_policy")
+ .enumType(MultiTableFailurePolicy.class)
+ .defaultValue(MultiTableFailurePolicy.FAIL_FAST)
+ .withDescription(
+ "Failure handling policy for multi-table jobs. "
+ + "FAIL_FAST aborts the whole job on the first table error. "
+ + "CONTINUE_OTHER_TABLES isolates failed tables and keeps healthy tables running when the error can be attributed to a single table.");
+
+ public static final Option<List<String>> MULTI_TABLE_INITIAL_FAILED_TABLES =
+ Options.key("seatunnel.multi_table.initial_failed_tables")
+ .listType()
+ .noDefaultValue()
+ .withDescription(
+ "Internal option used to propagate pre-runtime failed-table metadata into MultiTableSink.");
+
+ private MultiTableCommonOptions() {}
+}
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/MultiTableFailurePolicy.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/MultiTableFailurePolicy.java
new file mode 100644
index 0000000000..5e8847c946
--- /dev/null
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/options/MultiTableFailurePolicy.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.api.options;
+
+/** Failure handling policy for multi-table jobs. */
+public enum MultiTableFailurePolicy {
+ FAIL_FAST,
+ CONTINUE_OTHER_TABLES;
+
+ public boolean continueOtherTables() {
+ return this == CONTINUE_OTHER_TABLES;
+ }
+}
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSink.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSink.java
index ff13ceeb04..d26ae1b9a4 100644
--- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSink.java
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSink.java
@@ -18,6 +18,11 @@
package org.apache.seatunnel.api.sink.multitablesink;
import org.apache.seatunnel.api.common.JobContext;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.options.EnvCommonOptions;
+import org.apache.seatunnel.api.options.MultiTableCommonOptions;
+import org.apache.seatunnel.api.options.MultiTableFailurePolicy;
import org.apache.seatunnel.api.options.SinkConnectorCommonOptions;
import org.apache.seatunnel.api.serialization.DefaultSerializer;
import org.apache.seatunnel.api.serialization.Serializer;
@@ -31,6 +36,7 @@ import org.apache.seatunnel.api.table.catalog.TablePath;
import org.apache.seatunnel.api.table.factory.MultiTableFactoryContext;
import org.apache.seatunnel.api.table.schema.SchemaChangeType;
import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.common.constants.JobMode;
import lombok.Getter;
@@ -39,6 +45,7 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
@@ -63,6 +70,11 @@ public class MultiTableSink
@Getter private final Map<TablePath, SeaTunnelSink> sinks;
private final int replicaNum;
+ @Getter private final MultiTableFailurePolicy failurePolicy;
+ private final List<MultiTableFailedTable> initialFailedTables;
+ private final int tableRetryTimes;
+ private final int tableRetryIntervalSeconds;
+ private JobContext jobContext;
/**
* Constructs a MultiTableSink from the given factory context.
@@ -78,6 +90,18 @@ public class MultiTableSink
this.sinks = context.getSinks();
this.replicaNum =
context.getOptions().get(SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA);
+ this.failurePolicy =
+ context.getOptions().get(MultiTableCommonOptions.MULTI_TABLE_FAILURE_POLICY);
+ this.tableRetryTimes = context.getOptions().get(EnvCommonOptions.JOB_RETRY_TIMES);
+ this.tableRetryIntervalSeconds =
+ context.getOptions().get(EnvCommonOptions.JOB_RETRY_INTERVAL_SECONDS);
+ this.initialFailedTables =
+ new ArrayList<>(
+ MultiTableFailureHelper.getInitialFailedTables(context.getOptions()));
+ }
+
+ public List<MultiTableFailedTable> getInitialFailedTables() {
+ return Collections.unmodifiableList(initialFailedTables);
}
@Override
@@ -113,7 +137,15 @@ public class MultiTableSink
sinkWritersContext.put(SinkIdentifier.of(tableIdentifier, index), context);
}
}
- return new MultiTableSinkWriter(writers, replicaNum, sinkWritersContext);
+ return new MultiTableSinkWriter(
+ writers,
+ replicaNum,
+ sinkWritersContext,
+ failurePolicy,
+ getJobMode(),
+ initialFailedTables,
+ tableRetryTimes,
+ tableRetryIntervalSeconds);
}
/**
@@ -161,7 +193,15 @@ public class MultiTableSink
sinkWritersContext.put(sinkIdentifier, context);
}
}
- return new MultiTableSinkWriter(writers, replicaNum, sinkWritersContext);
+ return new MultiTableSinkWriter(
+ writers,
+ replicaNum,
+ sinkWritersContext,
+ failurePolicy,
+ getJobMode(),
+ initialFailedTables,
+ tableRetryTimes,
+ tableRetryIntervalSeconds);
}
@Override
@@ -270,9 +310,29 @@ public class MultiTableSink
@Override
public void setJobContext(JobContext jobContext) {
+ this.jobContext = jobContext;
sinks.values().forEach(sink -> sink.setJobContext(jobContext));
}
+ public void registerInitialFailedTables(Collection<MultiTableFailedTable> failedTables) {
+ if (failedTables == null || failedTables.isEmpty()) {
+ return;
+ }
+ Map<String, MultiTableFailedTable> mergedFailedTables = new LinkedHashMap<>();
+ initialFailedTables.forEach(
+ failedTable -> mergedFailedTables.put(failedTable.getTablePath(), failedTable));
+ failedTables.forEach(
+ failedTable -> mergedFailedTables.put(failedTable.getTablePath(), failedTable));
+ initialFailedTables.clear();
+ initialFailedTables.addAll(mergedFailedTables.values());
+ }
+
+ public void removeSink(TablePath tablePath) {
+ if (tablePath != null) {
+ sinks.remove(tablePath);
+ }
+ }
+
/**
* Always returns empty in multi-table context.
*
@@ -306,4 +366,10 @@ public class MultiTableSink
}
return Collections.emptyList();
}
+
+ private JobMode getJobMode() {
+ return jobContext == null || jobContext.getJobMode() == null
+ ? JobMode.BATCH
+ : jobContext.getJobMode();
+ }
}
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java
index 84837536fb..e5a9f952bb 100644
--- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java
@@ -17,6 +17,10 @@
package org.apache.seatunnel.api.sink.multitablesink;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
+import org.apache.seatunnel.api.options.MultiTableFailurePolicy;
import org.apache.seatunnel.api.sink.MultiTableResourceManager;
import org.apache.seatunnel.api.sink.SinkWriter;
import org.apache.seatunnel.api.sink.SupportMultiTableSinkWriter;
@@ -24,11 +28,14 @@ import org.apache.seatunnel.api.sink.SupportSchemaEvolutionSinkWriter;
import org.apache.seatunnel.api.table.schema.event.SchemaChangeEvent;
import org.apache.seatunnel.api.table.type.SeaTunnelRow;
import org.apache.seatunnel.api.tracing.MDCTracer;
+import org.apache.seatunnel.common.constants.JobMode;
import lombok.extern.slf4j.Slf4j;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -43,6 +50,7 @@ import java.util.concurrent.Future;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
/**
* A composite {@link SinkWriter} that distributes rows to multiple per-table sub-writers via async
@@ -65,15 +73,26 @@ public class MultiTableSinkWriter
private final Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters;
private final Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext;
- private final Map<String, Optional<Integer>> sinkPrimaryKeys = new HashMap<>();
+ private final ConcurrentMap<String, Optional<Integer>> sinkPrimaryKeys =
+ new ConcurrentHashMap<>();
private final List<ConcurrentMap<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>>>
sinkWritersWithIndex;
private final List<MultiTableWriterRunnable> runnable = new ArrayList<>();
private final Random random = new Random();
private final List<BlockingQueue<SeaTunnelRow>> blockingQueues = new ArrayList<>();
private final ExecutorService executorService;
+ private final MultiTableFailurePolicy failurePolicy;
+ private final JobMode jobMode;
+ private final int tableRetryTimes;
+ private final int tableRetryIntervalSeconds;
+ private final Map<String, List<SinkIdentifier>> sinkIdentifiersByTable =
+ new ConcurrentHashMap<>();
+ private final ConcurrentMap<String, MultiTableFailedTable> failedTables =
+ new ConcurrentHashMap<>();
private MultiTableResourceManager resourceManager;
private volatile boolean submitted = false;
+ private volatile Throwable fatalThrowable;
+ private volatile String fatalTableId;
/**
* Creates a MultiTableSinkWriter that distributes writes across multiple queues.
@@ -92,8 +111,86 @@ public class MultiTableSinkWriter
Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters,
int queueSize,
Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext) {
+ this(
+ sinkWriters,
+ queueSize,
+ sinkWritersContext,
+ MultiTableFailurePolicy.FAIL_FAST,
+ JobMode.BATCH,
+ Collections.emptyList(),
+ 0,
+ 0);
+ }
+
+ public MultiTableSinkWriter(
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters,
+ int queueSize,
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext,
+ MultiTableFailurePolicy failurePolicy,
+ JobMode jobMode) {
+ this(
+ sinkWriters,
+ queueSize,
+ sinkWritersContext,
+ failurePolicy,
+ jobMode,
+ Collections.emptyList(),
+ 0,
+ 0);
+ }
+
+ public MultiTableSinkWriter(
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters,
+ int queueSize,
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext,
+ MultiTableFailurePolicy failurePolicy,
+ JobMode jobMode,
+ int tableRetryTimes,
+ int tableRetryIntervalSeconds) {
+ this(
+ sinkWriters,
+ queueSize,
+ sinkWritersContext,
+ failurePolicy,
+ jobMode,
+ Collections.emptyList(),
+ tableRetryTimes,
+ tableRetryIntervalSeconds);
+ }
+
+ public MultiTableSinkWriter(
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters,
+ int queueSize,
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext,
+ MultiTableFailurePolicy failurePolicy,
+ JobMode jobMode,
+ Collection<MultiTableFailedTable> initialFailedTables) {
+ this(
+ sinkWriters,
+ queueSize,
+ sinkWritersContext,
+ failurePolicy,
+ jobMode,
+ initialFailedTables,
+ 0,
+ 0);
+ }
+
+ public MultiTableSinkWriter(
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters,
+ int queueSize,
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext,
+ MultiTableFailurePolicy failurePolicy,
+ JobMode jobMode,
+ Collection<MultiTableFailedTable> initialFailedTables,
+ int tableRetryTimes,
+ int tableRetryIntervalSeconds) {
this.sinkWriters = sinkWriters;
this.sinkWritersContext = sinkWritersContext;
+ this.failurePolicy = failurePolicy;
+ this.jobMode = jobMode;
+ this.tableRetryTimes = Math.max(0, tableRetryTimes);
+ this.tableRetryIntervalSeconds = Math.max(0, tableRetryIntervalSeconds);
AtomicInteger cnt = new AtomicInteger(0);
executorService =
MDCTracer.tracing(
@@ -125,15 +222,30 @@ public class MultiTableSinkWriter
tableIdWriterMap.put(
entry.getKey().getTableIdentifier(), entry.getValue());
sinkIdentifierMap.put(entry.getKey(), entry.getValue());
+ sinkIdentifiersByTable
+ .computeIfAbsent(
+ entry.getKey().getTableIdentifier(),
+ key ->
+ Collections.synchronizedList(
+ new ArrayList<>()))
+ .add(entry.getKey());
});
sinkWritersWithIndex.add(sinkIdentifierMap);
blockingQueues.add(queue);
- MultiTableWriterRunnable r = new MultiTableWriterRunnable(tableIdWriterMap, queue);
+ MultiTableWriterRunnable r =
+ new MultiTableWriterRunnable(
+ tableIdWriterMap,
+ queue,
+ failurePolicy.continueOtherTables(),
+ this::handleRuntimeTableFailure,
+ this.tableRetryTimes,
+ this.tableRetryIntervalSeconds);
runnable.add(r);
}
log.info("init multi table sink writer, queue size: {}", queueSize);
initResourceManager(queueSize);
+ registerInitialFailedTables(initialFailedTables);
}
/**
@@ -180,6 +292,9 @@ public class MultiTableSinkWriter
* fast.
*/
private void subSinkErrorCheck() {
+ if (fatalThrowable != null) {
+ throw new RuntimeException(buildFailureSummary(), fatalThrowable);
+ }
for (MultiTableWriterRunnable writerRunnable : runnable) {
if (writerRunnable.getThrowable() != null) {
throw new RuntimeException(
@@ -188,11 +303,21 @@ public class MultiTableSinkWriter
writerRunnable.getThrowable());
}
}
+ if (failurePolicy.continueOtherTables()
+ && sinkPrimaryKeys.isEmpty()
+ && !failedTables.isEmpty()) {
+ throw new RuntimeException(buildIsolatedFailureSummary(), getFirstFailureCause());
+ }
}
@Override
public void applySchemaChange(SchemaChangeEvent event) throws IOException {
subSinkErrorCheck();
+ String tableId = event.tablePath().getFullName();
+ if (isTableFailed(tableId)) {
+ log.warn("Skip schema change for failed table {}", tableId);
+ return;
+ }
for (int i = 0; i < sinkWritersWithIndex.size(); i++) {
for (Map.Entry<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriterEntry :
sinkWritersWithIndex.get(i).entrySet()) {
@@ -205,13 +330,34 @@ public class MultiTableSinkWriter
sinkWriterEntry.getKey().getTableIdentifier(),
sinkWriterEntry.getKey().getIndex());
synchronized (runnable.get(i)) {
- if (sinkWriterEntry.getValue()
- instanceof SupportSchemaEvolutionSinkWriter) {
- ((SupportSchemaEvolutionSinkWriter) sinkWriterEntry.getValue())
- .applySchemaChange(event);
- } else {
- // TODO remove deprecated method
- sinkWriterEntry.getValue().applySchemaChange(event);
+ try {
+ executeWithTableRetry(
+ sinkWriterEntry.getKey().getTableIdentifier(),
+ MultiTableFailurePhase.CHECKPOINT,
+ () -> {
+ if (sinkWriterEntry.getValue()
+ instanceof SupportSchemaEvolutionSinkWriter) {
+ ((SupportSchemaEvolutionSinkWriter)
+ sinkWriterEntry.getValue())
+ .applySchemaChange(event);
+ } else {
+ // TODO remove deprecated method
+ sinkWriterEntry.getValue().applySchemaChange(event);
+ }
+ return null;
+ });
+ } catch (InterruptedException error) {
+ Thread.currentThread().interrupt();
+ throwAsIOException(error);
+ } catch (Throwable error) {
+ if (failurePolicy.continueOtherTables()) {
+ handleTableFailure(
+ sinkWriterEntry.getKey().getTableIdentifier(),
+ MultiTableFailurePhase.CHECKPOINT,
+ error);
+ continue;
+ }
+ throwAsIOException(error);
}
}
log.info(
@@ -265,7 +411,12 @@ public class MultiTableSinkWriter
runnable.forEach(executorService::submit);
}
subSinkErrorCheck();
- Optional<Integer> primaryKey = sinkPrimaryKeys.get(element.getTableId());
+ String tableId = element.getTableId();
+ if (failurePolicy.continueOtherTables() && isTableFailed(tableId)) {
+ log.debug("Skip row for quarantined table {}", tableId);
+ return;
+ }
+ Optional<Integer> primaryKey = tableId == null ? null : sinkPrimaryKeys.get(tableId);
try {
if ((primaryKey == null && sinkPrimaryKeys.size() == 1)
|| (primaryKey != null && !primaryKey.isPresent())) {
@@ -275,8 +426,15 @@ public class MultiTableSinkWriter
subSinkErrorCheck();
}
} else if (primaryKey == null) {
- throw new RuntimeException(
- "multi table sink can not write table: " + element.getTableId());
+ if (failurePolicy.continueOtherTables()) {
+ handleTableFailure(
+ tableId,
+ MultiTableFailurePhase.RUNTIME_WRITE,
+ new IllegalStateException(
+ "No active sink writer found for table " + tableId));
+ return;
+ }
+ throw new RuntimeException("multi table sink can not write table: " + tableId);
} else {
Object object = element.getField(primaryKey.get());
int index = 0;
@@ -313,13 +471,36 @@ public class MultiTableSinkWriter
MultiTableState multiTableState = new MultiTableState(new HashMap<>());
for (int i = 0; i < sinkWritersWithIndex.size(); i++) {
for (Map.Entry<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriterEntry :
- sinkWritersWithIndex.get(i).entrySet()) {
+ new ArrayList<>(sinkWritersWithIndex.get(i).entrySet())) {
synchronized (runnable.get(i)) {
- List states = sinkWriterEntry.getValue().snapshotState(checkpointId);
- multiTableState.getStates().put(sinkWriterEntry.getKey(), states);
+ try {
+ List states =
+ executeWithTableRetry(
+ sinkWriterEntry.getKey().getTableIdentifier(),
+ MultiTableFailurePhase.CHECKPOINT,
+ () ->
+ sinkWriterEntry
+ .getValue()
+ .snapshotState(checkpointId));
+ multiTableState.getStates().put(sinkWriterEntry.getKey(), states);
+ } catch (InterruptedException error) {
+ Thread.currentThread().interrupt();
+ throwAsIOException(error);
+ } catch (Throwable error) {
+ if (failurePolicy.continueOtherTables()) {
+ handleTableFailure(
+ sinkWriterEntry.getKey().getTableIdentifier(),
+ MultiTableFailurePhase.CHECKPOINT,
+ error);
+ continue;
+ }
+ throwAsIOException(error);
+ }
}
}
}
+ waitRuntimeTableFailuresHandled();
+ subSinkErrorCheck();
multiTableStates.add(multiTableState);
return multiTableStates;
}
@@ -360,16 +541,37 @@ public class MultiTableSinkWriter
synchronized (runnable.get(subWriterIndex)) {
for (Map.Entry<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>>
sinkWriterEntry :
- sinkWritersWithIndex
- .get(subWriterIndex)
- .entrySet()) {
+ new ArrayList<>(
+ sinkWritersWithIndex
+ .get(subWriterIndex)
+ .entrySet())) {
Optional<?> commit;
try {
SinkWriter<SeaTunnelRow, ?, ?> sinkWriter =
sinkWriterEntry.getValue();
- commit = sinkWriter.prepareCommit(checkpointId);
- } catch (IOException e) {
- throw new RuntimeException(e);
+ commit =
+ executeWithTableRetry(
+ sinkWriterEntry
+ .getKey()
+ .getTableIdentifier(),
+ MultiTableFailurePhase.CHECKPOINT,
+ () ->
+ sinkWriter.prepareCommit(
+ checkpointId));
+ } catch (InterruptedException error) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(error);
+ } catch (Throwable error) {
+ if (failurePolicy.continueOtherTables()) {
+ handleTableFailure(
+ sinkWriterEntry
+ .getKey()
+ .getTableIdentifier(),
+ MultiTableFailurePhase.CHECKPOINT,
+ error);
+ continue;
+ }
+ throw new RuntimeException(error);
}
commit.ifPresent(
o ->
@@ -387,6 +589,8 @@ public class MultiTableSinkWriter
throw new RuntimeException(e);
}
}
+ waitRuntimeTableFailuresHandled();
+ subSinkErrorCheck();
if (multiTableCommitInfo.getCommitInfo().isEmpty()) {
return Optional.empty();
}
@@ -450,6 +654,7 @@ public class MultiTableSinkWriter
// The variables used in lambda expressions should be final or valid final, so they are
// modified to arrays
final Throwable[] firstE = {null};
+ boolean failedTableReportOnly = false;
try {
checkQueueRemain();
} catch (Exception e) {
@@ -480,7 +685,27 @@ public class MultiTableSinkWriter
} catch (Throwable e) {
log.error("close resourceManager error", e);
}
+ if (firstE[0] == null
+ && failurePolicy.continueOtherTables()
+ && jobMode == JobMode.BATCH
+ && !failedTables.isEmpty()) {
+ firstE[0] = getFirstFailureCause();
+ failedTableReportOnly = true;
+ }
if (firstE[0] != null) {
+ if (failurePolicy.continueOtherTables()
+ && jobMode == JobMode.BATCH
+ && !failedTables.isEmpty()) {
+ boolean isolatedFailureReport =
+ failedTableReportOnly
+ || MultiTableFailureHelper.isIsolatedFailure(
+ firstE[0].getMessage());
+ throw new IOException(
+ isolatedFailureReport
+ ? buildIsolatedFailureSummary()
+ : buildFailureSummary(),
+ firstE[0]);
+ }
throw new RuntimeException(firstE[0]);
}
}
@@ -495,14 +720,211 @@ public class MultiTableSinkWriter
*/
private void checkQueueRemain() {
try {
- for (BlockingQueue<SeaTunnelRow> blockingQueue : blockingQueues) {
- while (!blockingQueue.isEmpty()) {
- Thread.sleep(100);
- subSinkErrorCheck();
- }
+ while (hasPendingRuntimeWrites()) {
+ Thread.sleep(100);
+ subSinkErrorCheck();
}
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
+
+ private boolean hasPendingRuntimeWrites() {
+ for (BlockingQueue<SeaTunnelRow> blockingQueue : blockingQueues) {
+ if (!blockingQueue.isEmpty()) {
+ return true;
+ }
+ }
+ for (MultiTableWriterRunnable writerRunnable : runnable) {
+ if (writerRunnable.isProcessingRow() || writerRunnable.isHandlingTableFailure()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private void handleRuntimeTableFailure(String tableId, Throwable error) {
+ if (failurePolicy.continueOtherTables()) {
+ handleTableFailure(tableId, MultiTableFailurePhase.RUNTIME_WRITE, error);
+ return;
+ }
+ fatalTableId = tableId;
+ fatalThrowable = error;
+ }
+
+ private synchronized void handleTableFailure(
+ String tableId, MultiTableFailurePhase phase, Throwable error) {
+ if (tableId == null || tableId.trim().isEmpty()) {
+ fatalTableId = tableId;
+ fatalThrowable = error;
+ return;
+ }
+ if (failedTables.containsKey(tableId)) {
+ return;
+ }
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(tableId, phase, "MultiTableSink", error);
+ registerFailedTable(failedTable, true);
+ }
+
+ private <T> T executeWithTableRetry(
+ String tableId, MultiTableFailurePhase phase, TableOperation<T> operation)
+ throws Throwable {
+ int retriedTimes = 0;
+ while (true) {
+ try {
+ return operation.execute();
+ } catch (InterruptedException error) {
+ Thread.currentThread().interrupt();
+ throw error;
+ } catch (Throwable error) {
+ if (!failurePolicy.continueOtherTables() || retriedTimes >= tableRetryTimes) {
+ throw error;
+ }
+ retriedTimes++;
+ log.warn(
+ "Retry multi-table sink operation for table {} in phase {}, attempt {}/{}",
+ tableId,
+ phase,
+ retriedTimes,
+ tableRetryTimes,
+ error);
+ waitBeforeTableRetry();
+ }
+ }
+ }
+
+ private void waitBeforeTableRetry() throws InterruptedException {
+ if (tableRetryIntervalSeconds <= 0) {
+ return;
+ }
+ try {
+ TimeUnit.SECONDS.sleep(tableRetryIntervalSeconds);
+ } catch (InterruptedException error) {
+ Thread.currentThread().interrupt();
+ throw error;
+ }
+ }
+
+ private void waitRuntimeTableFailuresHandled() {
+ try {
+ boolean handlingFailure = true;
+ while (handlingFailure) {
+ handlingFailure = false;
+ for (MultiTableWriterRunnable writerRunnable : runnable) {
+ if (writerRunnable.isHandlingTableFailure()) {
+ handlingFailure = true;
+ break;
+ }
+ }
+ if (handlingFailure) {
+ Thread.sleep(10);
+ }
+ }
+ } catch (InterruptedException error) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(error);
+ }
+ }
+
+ private boolean isTableFailed(String tableId) {
+ return tableId != null && failedTables.containsKey(tableId);
+ }
+
+ private Throwable getFirstFailureCause() {
+ return failedTables.values().stream()
+ .map(MultiTableFailedTable::getCause)
+ .filter(cause -> cause != null)
+ .findFirst()
+ .orElseGet(
+ () ->
+ fatalThrowable != null
+ ? fatalThrowable
+ : new IllegalStateException(buildFailureSummary()));
+ }
+
+ private String buildFailureSummary() {
+ List<MultiTableFailedTable> failures =
+ failedTables.values().stream()
+ .sorted(
+ (left, right) ->
+ left.getTablePath().compareTo(right.getTablePath()))
+ .collect(Collectors.toList());
+ String title =
+ fatalTableId == null
+ ? "Failed tables were isolated in multi-table sink."
+ : String.format(
+ "Failed table '%s' caused multi-table sink to abort.",
+ fatalTableId);
+ return MultiTableFailureHelper.formatFailedTableSummary(title, failures);
+ }
+
+ private String buildIsolatedFailureSummary() {
+ return MultiTableFailureHelper.withIsolatedFailureMarker(buildFailureSummary());
+ }
+
+ private void throwAsIOException(Throwable error) throws IOException {
+ if (error instanceof IOException) {
+ throw (IOException) error;
+ }
+ if (error instanceof RuntimeException) {
+ throw (RuntimeException) error;
+ }
+ throw new IOException(error);
+ }
+
+ private void registerInitialFailedTables(
+ Collection<MultiTableFailedTable> initialFailedTables) {
+ if (initialFailedTables == null || initialFailedTables.isEmpty()) {
+ return;
+ }
+ initialFailedTables.forEach(failedTable -> registerFailedTable(failedTable, false));
+ }
+
+ private synchronized void registerFailedTable(
+ MultiTableFailedTable failedTable, boolean printLog) {
+ String tableId = failedTable.getTablePath();
+ if (tableId == null || tableId.trim().isEmpty() || failedTables.containsKey(tableId)) {
+ return;
+ }
+ failedTables.put(tableId, failedTable);
+ if (printLog) {
+ log.warn(
+ "Quarantine failed table in multi-table sink: {}",
+ MultiTableFailureHelper.formatFailedTableLine(failedTable),
+ failedTable.getCause());
+ }
+ removeTableWriters(tableId);
+ }
+
+ private void removeTableWriters(String tableId) {
+ sinkPrimaryKeys.remove(tableId);
+ List<SinkIdentifier> sinkIdentifiers =
+ sinkIdentifiersByTable.getOrDefault(tableId, Collections.emptyList());
+ for (SinkIdentifier sinkIdentifier : sinkIdentifiers) {
+ sinkWriters.remove(sinkIdentifier);
+ for (int i = 0; i < sinkWritersWithIndex.size(); i++) {
+ synchronized (runnable.get(i)) {
+ SinkWriter<SeaTunnelRow, ?, ?> removedWriter =
+ sinkWritersWithIndex.get(i).remove(sinkIdentifier);
+ runnable.get(i).removeTableWriter(tableId);
+ if (removedWriter != null) {
+ try {
+ removedWriter.close();
+ } catch (Throwable closeError) {
+ log.warn(
+ "Close quarantined writer failed for table {}",
+ tableId,
+ closeError);
+ }
+ }
+ }
+ }
+ }
+ }
+
+ @FunctionalInterface
+ private interface TableOperation<T> {
+ T execute() throws Throwable;
+ }
}
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableWriterRunnable.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableWriterRunnable.java
index cbb0ff0801..0e5fe4264d 100644
--- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableWriterRunnable.java
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableWriterRunnable.java
@@ -25,69 +25,193 @@ import lombok.extern.slf4j.Slf4j;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.TimeUnit;
+import java.util.function.BiConsumer;
@Slf4j
public class MultiTableWriterRunnable implements Runnable {
private final Map<String, SinkWriter<SeaTunnelRow, ?, ?>> tableIdWriterMap;
private final BlockingQueue<SeaTunnelRow> queue;
+ private final boolean allowSingleWriterFallback;
+ private final boolean continueOnTableFailure;
+ private final BiConsumer<String, Throwable> failureHandler;
+ private final int tableRetryTimes;
+ private final int tableRetryIntervalSeconds;
private volatile Throwable throwable;
private volatile String currentTableId;
+ private volatile boolean processingRow;
+ private volatile boolean handlingTableFailure;
public MultiTableWriterRunnable(
Map<String, SinkWriter<SeaTunnelRow, ?, ?>> tableIdWriterMap,
BlockingQueue<SeaTunnelRow> queue) {
+ this(tableIdWriterMap, queue, false, (tableId, error) -> {});
+ }
+
+ public MultiTableWriterRunnable(
+ Map<String, SinkWriter<SeaTunnelRow, ?, ?>> tableIdWriterMap,
+ BlockingQueue<SeaTunnelRow> queue,
+ boolean continueOnTableFailure,
+ BiConsumer<String, Throwable> failureHandler) {
+ this(tableIdWriterMap, queue, continueOnTableFailure, failureHandler, 0, 0);
+ }
+
+ public MultiTableWriterRunnable(
+ Map<String, SinkWriter<SeaTunnelRow, ?, ?>> tableIdWriterMap,
+ BlockingQueue<SeaTunnelRow> queue,
+ boolean continueOnTableFailure,
+ BiConsumer<String, Throwable> failureHandler,
+ int tableRetryTimes,
+ int tableRetryIntervalSeconds) {
this.tableIdWriterMap = tableIdWriterMap;
this.queue = queue;
+ this.allowSingleWriterFallback = tableIdWriterMap.size() == 1;
+ this.continueOnTableFailure = continueOnTableFailure;
+ this.failureHandler = failureHandler;
+ this.tableRetryTimes = Math.max(0, tableRetryTimes);
+ this.tableRetryIntervalSeconds = Math.max(0, tableRetryIntervalSeconds);
}
@Override
public void run() {
while (true) {
SeaTunnelRow row = null;
+ TableFailure tableFailure = null;
try {
row = queue.poll(100, TimeUnit.MILLISECONDS);
if (row == null) {
continue;
}
+ processingRow = true;
// control rows used for schema evolution / coordination
// are represented as SeaTunnelRow with zero fields (arity == 0)
if (row.getArity() == 0) {
log.debug(
"Skip control SeaTunnelRow with zero arity in MultiTableWriterRunnable: {}",
row);
+ processingRow = false;
continue;
}
- SinkWriter<SeaTunnelRow, ?, ?> writer = tableIdWriterMap.get(row.getTableId());
- if (writer == null) {
- if (tableIdWriterMap.size() == 1) {
- writer = tableIdWriterMap.values().stream().findFirst().get();
- currentTableId = tableIdWriterMap.keySet().stream().findFirst().get();
+ synchronized (this) {
+ SinkWriter<SeaTunnelRow, ?, ?> writer = tableIdWriterMap.get(row.getTableId());
+ if (writer == null) {
+ // Single-table jobs may still emit rewritten/non-canonical table ids.
+ // Keep the historical sole-writer fallback only for runnables that
+ // started with one writer so quarantined multi-table rows are not rerouted.
+ if (allowSingleWriterFallback && tableIdWriterMap.size() == 1) {
+ writer = tableIdWriterMap.values().stream().findFirst().get();
+ currentTableId = tableIdWriterMap.keySet().stream().findFirst().get();
+ } else if (continueOnTableFailure) {
+ log.debug("Skip row for quarantined table {}", row.getTableId());
+ processingRow = false;
+ continue;
+ } else {
+ throw new RuntimeException(
+ "MultiTableWriterRunnable can't find writer for tableId: "
+ + row.getTableId());
+ }
} else {
- throw new RuntimeException(
- "MultiTableWriterRunnable can't find writer for tableId: "
- + row.getTableId());
+ currentTableId = row.getTableId();
+ }
+ try {
+ writeWithRetry(writer, row, currentTableId);
+ processingRow = false;
+ } catch (InterruptedException e) {
+ processingRow = false;
+ throw e;
+ } catch (Throwable e) {
+ tableFailure = handleWriteFailure(row, e);
+ if (tableFailure == null) {
+ processingRow = false;
+ break;
+ }
}
- } else {
- currentTableId = row.getTableId();
}
- synchronized (this) {
- writer.write(row);
+ if (tableFailure != null) {
+ if (notifyTableFailure(tableFailure)) {
+ continue;
+ }
+ break;
}
} catch (InterruptedException e) {
// When the job finished, the thread will be interrupted, so we ignore this
// exception.
- throwable = e;
+ processingRow = false;
break;
} catch (Throwable e) {
- log.error(
- String.format("MultiTableWriterRunnable error when write row %s", row), e);
- throwable = e;
+ tableFailure = handleWriteFailure(row, e);
+ if (tableFailure != null && notifyTableFailure(tableFailure)) {
+ continue;
+ }
+ processingRow = false;
break;
}
}
}
+ private TableFailure handleWriteFailure(SeaTunnelRow row, Throwable error) {
+ log.error(String.format("MultiTableWriterRunnable error when write row %s", row), error);
+ String failedTableId =
+ currentTableId != null ? currentTableId : row == null ? null : row.getTableId();
+ if (continueOnTableFailure && failedTableId != null && !failedTableId.trim().isEmpty()) {
+ removeTableWriter(failedTableId);
+ currentTableId = null;
+ handlingTableFailure = true;
+ return new TableFailure(failedTableId, error);
+ }
+ throwable = error;
+ return null;
+ }
+
+ private boolean notifyTableFailure(TableFailure tableFailure) {
+ try {
+ failureHandler.accept(tableFailure.tableId, tableFailure.error);
+ return true;
+ } catch (Throwable error) {
+ throwable = error;
+ return false;
+ } finally {
+ handlingTableFailure = false;
+ processingRow = false;
+ }
+ }
+
+ private void writeWithRetry(
+ SinkWriter<SeaTunnelRow, ?, ?> writer, SeaTunnelRow row, String tableId)
+ throws Throwable {
+ int retriedTimes = 0;
+ while (true) {
+ try {
+ writer.write(row);
+ return;
+ } catch (Throwable error) {
+ if (!continueOnTableFailure || retriedTimes >= tableRetryTimes) {
+ throw error;
+ }
+ retriedTimes++;
+ log.warn(
+ "Retry multi-table sink write for table {}, attempt {}/{}",
+ tableId,
+ retriedTimes,
+ tableRetryTimes,
+ error);
+ waitBeforeRetry();
+ }
+ }
+ }
+
+ private void waitBeforeRetry() throws InterruptedException {
+ if (tableRetryIntervalSeconds <= 0) {
+ return;
+ }
+ try {
+ TimeUnit.SECONDS.sleep(tableRetryIntervalSeconds);
+ } catch (InterruptedException interruptedException) {
+ Thread.currentThread().interrupt();
+ throw interruptedException;
+ }
+ }
+
public Throwable getThrowable() {
return throwable;
}
@@ -95,4 +219,26 @@ public class MultiTableWriterRunnable implements Runnable {
public String getCurrentTableId() {
return currentTableId;
}
+
+ public boolean isProcessingRow() {
+ return processingRow;
+ }
+
+ public boolean isHandlingTableFailure() {
+ return handlingTableFailure;
+ }
+
+ public synchronized void removeTableWriter(String tableId) {
+ tableIdWriterMap.remove(tableId);
+ }
+
+ private static class TableFailure {
+ private final String tableId;
+ private final Throwable error;
+
+ private TableFailure(String tableId, Throwable error) {
+ this.tableId = tableId;
+ this.error = error;
+ }
+ }
}
diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/Catalog.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/Catalog.java
index ebc7d183b1..532bc6b673 100644
--- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/Catalog.java
+++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/Catalog.java
@@ -19,6 +19,9 @@ package org.apache.seatunnel.api.table.catalog;
import org.apache.seatunnel.shade.org.apache.commons.lang3.StringUtils;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.options.ConnectorCommonOptions;
import org.apache.seatunnel.api.table.catalog.exception.CatalogException;
@@ -31,6 +34,9 @@ import org.apache.seatunnel.common.exception.CommonError;
import org.apache.seatunnel.common.exception.CommonErrorCode;
import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
@@ -48,6 +54,8 @@ import java.util.stream.Collectors;
*/
public interface Catalog extends AutoCloseable {
+ Logger LOG = LoggerFactory.getLogger(Catalog.class);
+
default Optional<Factory> getFactory() {
return Optional.empty();
}
@@ -154,7 +162,7 @@ public interface Catalog extends AutoCloseable {
if (tableNames != null && !tableNames.isEmpty()) {
Iterator<TablePath> tablePaths =
tableNames.stream().map(TablePath::of).filter(this::tableExists).iterator();
- return buildCatalogTablesWithErrorCheck(tablePaths);
+ return buildCatalogTablesWithErrorCheck(tablePaths, config);
}
// Get the list of table pattern
@@ -184,7 +192,7 @@ public interface Catalog extends AutoCloseable {
.matches())
.collect(Collectors.toList()));
}
- return buildCatalogTablesWithErrorCheck(tablePaths.iterator());
+ return buildCatalogTablesWithErrorCheck(tablePaths.iterator(), config);
}
default List<TablePath> listTablePaths(String databaseName)
@@ -204,20 +212,39 @@ public interface Catalog extends AutoCloseable {
}
default List<CatalogTable> buildCatalogTablesWithErrorCheck(Iterator<TablePath> tablePaths) {
+ return buildCatalogTablesWithErrorCheck(tablePaths, null);
+ }
+
+ default List<CatalogTable> buildCatalogTablesWithErrorCheck(
+ Iterator<TablePath> tablePaths, ReadonlyConfig config) {
Map<String, Map<String, String>> unsupportedTable = new LinkedHashMap<>();
List<CatalogTable> catalogTables = new ArrayList<>();
+ boolean continueOtherTables = MultiTableFailureHelper.shouldContinueOtherTables(config);
while (tablePaths.hasNext()) {
+ TablePath tablePath = tablePaths.next();
try {
- catalogTables.add(getTable(tablePaths.next()));
+ catalogTables.add(getTable(tablePath));
} catch (SeaTunnelRuntimeException e) {
if (e.getSeaTunnelErrorCode()
.equals(CommonErrorCode.GET_CATALOG_TABLE_WITH_UNSUPPORTED_TYPE_ERROR)) {
- unsupportedTable.put(
- e.getParams().get("tableName"),
- e.getParamsValueAsMap("fieldWithDataTypes"));
+ if (continueOtherTables) {
+ logSkipFailedTable(tablePath, e);
+ } else {
+ unsupportedTable.put(
+ e.getParams().get("tableName"),
+ e.getParamsValueAsMap("fieldWithDataTypes"));
+ }
+ } else if (continueOtherTables) {
+ logSkipFailedTable(tablePath, e);
} else {
throw e;
}
+ } catch (Exception e) {
+ if (continueOtherTables) {
+ logSkipFailedTable(tablePath, e);
+ } else {
+ throw wrapThrowable(e);
+ }
}
}
if (!unsupportedTable.isEmpty()) {
@@ -250,6 +277,24 @@ public interface Catalog extends AutoCloseable {
}
}
+ default void logSkipFailedTable(TablePath tablePath, Throwable error) {
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ tablePath.getFullName(), MultiTableFailurePhase.DISCOVERY, name(), error);
+ MultiTableFailureHelper.recordFailedTable(failedTable);
+ LOG.warn(
+ "Skip failed table in discovery: {}",
+ MultiTableFailureHelper.formatFailedTableLine(failedTable),
+ error);
+ }
+
+ default RuntimeException wrapThrowable(Throwable error) {
+ if (error instanceof RuntimeException) {
+ return (RuntimeException) error;
+ }
+ return new RuntimeException(error);
+ }
+
/**
* Create a new table in this catalog.
*
diff --git a/seatunnel-api/src/test/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriterTest.java b/seatunnel-api/src/test/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriterTest.java
index 86722eb246..c068921f0f 100644
--- a/seatunnel-api/src/test/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriterTest.java
+++ b/seatunnel-api/src/test/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriterTest.java
@@ -18,14 +18,20 @@
package org.apache.seatunnel.api.sink.multitablesink;
import org.apache.seatunnel.api.common.metrics.MetricsContext;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.event.DefaultEventProcessor;
import org.apache.seatunnel.api.event.EventListener;
+import org.apache.seatunnel.api.options.MultiTableFailurePolicy;
import org.apache.seatunnel.api.serialization.DefaultSerializer;
import org.apache.seatunnel.api.sink.SinkWriter;
import org.apache.seatunnel.api.sink.SupportMultiTableSinkWriter;
import org.apache.seatunnel.api.table.catalog.TablePath;
import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.common.constants.JobMode;
+import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import lombok.AllArgsConstructor;
@@ -33,10 +39,17 @@ import lombok.Data;
import java.io.IOException;
import java.io.Serializable;
+import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
public class MultiTableSinkWriterTest {
@@ -62,6 +75,396 @@ public class MultiTableSinkWriterTest {
}
}
+ @Test
+ public void testContinueOtherTablesKeepsHealthyTableRunning() throws IOException {
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters = new HashMap<>();
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext = new HashMap<>();
+ RecordingSinkWriter failedWriter = new RecordingSinkWriter(true);
+ RecordingSinkWriter healthyWriter = new RecordingSinkWriter(false);
+ SinkIdentifier failedIdentifier = SinkIdentifier.of("test.failed", 0);
+ SinkIdentifier healthyIdentifier = SinkIdentifier.of("test.healthy", 1);
+ sinkWriters.put(failedIdentifier, failedWriter);
+ sinkWriters.put(healthyIdentifier, healthyWriter);
+ sinkWritersContext.put(failedIdentifier, new TestSinkWriterContext());
+ sinkWritersContext.put(healthyIdentifier, new TestSinkWriterContext());
+
+ MultiTableSinkWriter multiTableSinkWriter =
+ new MultiTableSinkWriter(
+ sinkWriters,
+ 2,
+ sinkWritersContext,
+ MultiTableFailurePolicy.CONTINUE_OTHER_TABLES,
+ JobMode.BATCH);
+
+ multiTableSinkWriter.write(buildRow("test.failed", 0));
+ multiTableSinkWriter.write(buildRow("test.healthy", 1));
+
+ Optional<MultiTableCommitInfo> commitInfo = multiTableSinkWriter.prepareCommit(1L);
+ Assertions.assertTrue(commitInfo.isPresent());
+ Assertions.assertEquals(1, commitInfo.get().getCommitInfo().size());
+ Assertions.assertTrue(
+ commitInfo.get().getCommitInfo().keySet().stream()
+ .allMatch(
+ identifier ->
+ "test.healthy".equals(identifier.getTableIdentifier())));
+ Assertions.assertEquals(1, healthyWriter.getWriteCount());
+ Assertions.assertEquals(1, failedWriter.getWriteCount());
+
+ IOException closeException =
+ Assertions.assertThrows(IOException.class, multiTableSinkWriter::close);
+ Assertions.assertTrue(closeException.getMessage().contains("test.failed"));
+ }
+
+ @Test
+ public void testInitialFailedTableIsSkippedAndReported() throws IOException {
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters = new HashMap<>();
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext = new HashMap<>();
+ RecordingSinkWriter skippedWriter = new RecordingSinkWriter(false);
+ RecordingSinkWriter healthyWriter = new RecordingSinkWriter(false);
+ SinkIdentifier skippedIdentifier = SinkIdentifier.of("test.skipped", 0);
+ SinkIdentifier healthyIdentifier = SinkIdentifier.of("test.healthy", 1);
+ sinkWriters.put(skippedIdentifier, skippedWriter);
+ sinkWriters.put(healthyIdentifier, healthyWriter);
+ sinkWritersContext.put(skippedIdentifier, new TestSinkWriterContext());
+ sinkWritersContext.put(healthyIdentifier, new TestSinkWriterContext());
+
+ MultiTableFailedTable initialFailedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ "test.skipped",
+ MultiTableFailurePhase.SINK_INIT,
+ "console",
+ new RuntimeException("startup failure"));
+ MultiTableSinkWriter multiTableSinkWriter =
+ new MultiTableSinkWriter(
+ sinkWriters,
+ 2,
+ sinkWritersContext,
+ MultiTableFailurePolicy.CONTINUE_OTHER_TABLES,
+ JobMode.BATCH,
+ Collections.singletonList(initialFailedTable));
+
+ multiTableSinkWriter.write(buildRow("test.skipped", 0));
+ multiTableSinkWriter.write(buildRow("test.healthy", 1));
+
+ Optional<MultiTableCommitInfo> commitInfo = multiTableSinkWriter.prepareCommit(1L);
+ Assertions.assertTrue(commitInfo.isPresent());
+ Assertions.assertEquals(1, commitInfo.get().getCommitInfo().size());
+ Assertions.assertEquals(0, skippedWriter.getWriteCount());
+ Assertions.assertEquals(1, healthyWriter.getWriteCount());
+
+ IOException closeException =
+ Assertions.assertThrows(IOException.class, multiTableSinkWriter::close);
+ Assertions.assertTrue(closeException.getMessage().contains("test.skipped"));
+ Assertions.assertTrue(closeException.getMessage().contains("startup failure"));
+ }
+
+ @Test
+ public void testRuntimeWriteRetriesFailedTableBeforeIsolation() throws IOException {
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters = new HashMap<>();
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext = new HashMap<>();
+ RecordingSinkWriter failedThenRecoveredWriter = new RetryableWriteSinkWriter(2);
+ RecordingSinkWriter healthyWriter = new RecordingSinkWriter(false);
+ SinkIdentifier failedIdentifier = SinkIdentifier.of("test.failed", 0);
+ SinkIdentifier healthyIdentifier = SinkIdentifier.of("test.healthy", 1);
+ sinkWriters.put(failedIdentifier, failedThenRecoveredWriter);
+ sinkWriters.put(healthyIdentifier, healthyWriter);
+ sinkWritersContext.put(failedIdentifier, new TestSinkWriterContext());
+ sinkWritersContext.put(healthyIdentifier, new TestSinkWriterContext());
+
+ MultiTableSinkWriter multiTableSinkWriter =
+ new MultiTableSinkWriter(
+ sinkWriters,
+ 2,
+ sinkWritersContext,
+ MultiTableFailurePolicy.CONTINUE_OTHER_TABLES,
+ JobMode.BATCH,
+ 2,
+ 0);
+
+ multiTableSinkWriter.write(buildRow("test.failed", 0));
+ multiTableSinkWriter.write(buildRow("test.healthy", 1));
+
+ Optional<MultiTableCommitInfo> commitInfo = multiTableSinkWriter.prepareCommit(1L);
+ Assertions.assertTrue(commitInfo.isPresent());
+ Assertions.assertEquals(2, commitInfo.get().getCommitInfo().size());
+ Assertions.assertEquals(3, failedThenRecoveredWriter.getWriteCount());
+ Assertions.assertEquals(1, healthyWriter.getWriteCount());
+ Assertions.assertDoesNotThrow(multiTableSinkWriter::close);
+ }
+
+ @Test
+ public void testRuntimeWriteIsolatesFailedTableAfterRetryExhausted() throws IOException {
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters = new HashMap<>();
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext = new HashMap<>();
+ RecordingSinkWriter failedWriter = new RecordingSinkWriter(true);
+ RecordingSinkWriter healthyWriter = new RecordingSinkWriter(false);
+ SinkIdentifier failedIdentifier = SinkIdentifier.of("test.failed", 0);
+ SinkIdentifier healthyIdentifier = SinkIdentifier.of("test.healthy", 1);
+ sinkWriters.put(failedIdentifier, failedWriter);
+ sinkWriters.put(healthyIdentifier, healthyWriter);
+ sinkWritersContext.put(failedIdentifier, new TestSinkWriterContext());
+ sinkWritersContext.put(healthyIdentifier, new TestSinkWriterContext());
+
+ MultiTableSinkWriter multiTableSinkWriter =
+ new MultiTableSinkWriter(
+ sinkWriters,
+ 2,
+ sinkWritersContext,
+ MultiTableFailurePolicy.CONTINUE_OTHER_TABLES,
+ JobMode.BATCH,
+ 2,
+ 0);
+
+ multiTableSinkWriter.write(buildRow("test.failed", 0));
+ multiTableSinkWriter.write(buildRow("test.healthy", 1));
+
+ Optional<MultiTableCommitInfo> commitInfo = multiTableSinkWriter.prepareCommit(1L);
+ Assertions.assertTrue(commitInfo.isPresent());
+ Assertions.assertEquals(1, commitInfo.get().getCommitInfo().size());
+ Assertions.assertEquals(3, failedWriter.getWriteCount());
+ Assertions.assertEquals(1, healthyWriter.getWriteCount());
+
+ IOException closeException =
+ Assertions.assertThrows(IOException.class, multiTableSinkWriter::close);
+ Assertions.assertTrue(closeException.getMessage().contains("test.failed"));
+ Assertions.assertTrue(
+ MultiTableFailureHelper.isIsolatedFailure(closeException.getMessage()));
+ }
+
+ @Test
+ public void testRuntimeWriteZeroRetryKeepsImmediateIsolation() throws IOException {
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters = new HashMap<>();
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext = new HashMap<>();
+ RecordingSinkWriter failedWriter = new RecordingSinkWriter(true);
+ SinkIdentifier failedIdentifier = SinkIdentifier.of("test.failed", 0);
+ sinkWriters.put(failedIdentifier, failedWriter);
+ sinkWritersContext.put(failedIdentifier, new TestSinkWriterContext());
+
+ MultiTableSinkWriter multiTableSinkWriter =
+ new MultiTableSinkWriter(
+ sinkWriters,
+ 1,
+ sinkWritersContext,
+ MultiTableFailurePolicy.CONTINUE_OTHER_TABLES,
+ JobMode.BATCH,
+ 0,
+ 0);
+
+ multiTableSinkWriter.write(buildRow("test.failed", 0));
+
+ RuntimeException prepareCommitException =
+ Assertions.assertThrows(
+ RuntimeException.class, () -> multiTableSinkWriter.prepareCommit(1L));
+ Assertions.assertTrue(
+ MultiTableFailureHelper.isIsolatedFailure(prepareCommitException.getMessage()));
+ Assertions.assertEquals(1, failedWriter.getWriteCount());
+ IOException closeException =
+ Assertions.assertThrows(IOException.class, multiTableSinkWriter::close);
+ Assertions.assertTrue(
+ MultiTableFailureHelper.isIsolatedFailure(closeException.getMessage()));
+ }
+
+ @Test
+ public void testFailFastDoesNotUseTableRetry() throws IOException {
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters = new HashMap<>();
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext = new HashMap<>();
+ RecordingSinkWriter failedWriter = new RecordingSinkWriter(true);
+ SinkIdentifier failedIdentifier = SinkIdentifier.of("test.failed", 0);
+ sinkWriters.put(failedIdentifier, failedWriter);
+ sinkWritersContext.put(failedIdentifier, new TestSinkWriterContext());
+
+ MultiTableSinkWriter multiTableSinkWriter =
+ new MultiTableSinkWriter(
+ sinkWriters,
+ 1,
+ sinkWritersContext,
+ MultiTableFailurePolicy.FAIL_FAST,
+ JobMode.BATCH,
+ 2,
+ 0);
+
+ multiTableSinkWriter.write(buildRow("test.failed", 0));
+
+ Assertions.assertThrows(
+ RuntimeException.class, () -> multiTableSinkWriter.prepareCommit(1L));
+ Assertions.assertEquals(1, failedWriter.getWriteCount());
+ multiTableSinkWriter.close();
+ }
+
+ @Test
+ public void testPrepareCommitRetriesBeforeIsolation() throws IOException {
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters = new HashMap<>();
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext = new HashMap<>();
+ PrepareCommitRetrySinkWriter retryWriter = new PrepareCommitRetrySinkWriter(1);
+ SinkIdentifier sinkIdentifier = SinkIdentifier.of("test.retry", 0);
+ sinkWriters.put(sinkIdentifier, retryWriter);
+ sinkWritersContext.put(sinkIdentifier, new TestSinkWriterContext());
+
+ MultiTableSinkWriter multiTableSinkWriter =
+ new MultiTableSinkWriter(
+ sinkWriters,
+ 1,
+ sinkWritersContext,
+ MultiTableFailurePolicy.CONTINUE_OTHER_TABLES,
+ JobMode.BATCH,
+ 1,
+ 0);
+
+ Optional<MultiTableCommitInfo> commitInfo = multiTableSinkWriter.prepareCommit(1L);
+
+ Assertions.assertTrue(commitInfo.isPresent());
+ Assertions.assertEquals(1, commitInfo.get().getCommitInfo().size());
+ Assertions.assertEquals(2, retryWriter.getPrepareCommitCount());
+ Assertions.assertDoesNotThrow(multiTableSinkWriter::close);
+ }
+
+ @Test
+ public void testSnapshotStateRetriesBeforeIsolation() throws IOException {
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters = new HashMap<>();
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext = new HashMap<>();
+ SnapshotRetrySinkWriter retryWriter = new SnapshotRetrySinkWriter(1);
+ SinkIdentifier sinkIdentifier = SinkIdentifier.of("test.retry", 0);
+ sinkWriters.put(sinkIdentifier, retryWriter);
+ sinkWritersContext.put(sinkIdentifier, new TestSinkWriterContext());
+
+ MultiTableSinkWriter multiTableSinkWriter =
+ new MultiTableSinkWriter(
+ sinkWriters,
+ 1,
+ sinkWritersContext,
+ MultiTableFailurePolicy.CONTINUE_OTHER_TABLES,
+ JobMode.BATCH,
+ 1,
+ 0);
+
+ List<MultiTableState> states = multiTableSinkWriter.snapshotState(1L);
+
+ Assertions.assertEquals(1, states.size());
+ Assertions.assertEquals(1, retryWriter.getSnapshotCount());
+ Assertions.assertEquals(2, retryWriter.getSnapshotAttemptCount());
+ Assertions.assertDoesNotThrow(multiTableSinkWriter::close);
+ }
+
+ @Test
+ public void testIsolatedFailureMarkerRecognition() {
+ String message =
+ MultiTableFailureHelper.withIsolatedFailureMarker(
+ "Failed tables were isolated in multi-table sink.");
+
+ Assertions.assertTrue(MultiTableFailureHelper.isIsolatedFailure(message));
+ Assertions.assertFalse(MultiTableFailureHelper.isIsolatedFailure(null));
+ Assertions.assertFalse(MultiTableFailureHelper.isIsolatedFailure("regular failure"));
+ }
+
+ @Test
+ public void testSingleWriterFallbackAcceptsExplicitTableId() {
+ Map<String, SinkWriter<SeaTunnelRow, ?, ?>> tableIdWriterMap = new HashMap<>();
+ RecordingSinkWriter onlyWriter = new RecordingSinkWriter(false, true);
+ BlockingQueue<SeaTunnelRow> queue = new LinkedBlockingQueue<>(1);
+ tableIdWriterMap.put("http", onlyWriter);
+ queue.add(buildRow("Optional[http]", 1));
+
+ MultiTableWriterRunnable runnable = new MultiTableWriterRunnable(tableIdWriterMap, queue);
+ runnable.run();
+
+ Assertions.assertNull(runnable.getThrowable());
+ Assertions.assertEquals(1, onlyWriter.getWriteCount());
+ Assertions.assertEquals("http", runnable.getCurrentTableId());
+ }
+
+ @Test
+ public void testRunnableSelectsWriterUnderLock() {
+ GuardedWriterMap tableIdWriterMap = new GuardedWriterMap();
+ RecordingSinkWriter writer = new RecordingSinkWriter(false, true);
+ BlockingQueue<SeaTunnelRow> queue = new LinkedBlockingQueue<>(1);
+ tableIdWriterMap.put("test.table", writer);
+ queue.add(buildRow("test.table", 1));
+
+ MultiTableWriterRunnable runnable = new MultiTableWriterRunnable(tableIdWriterMap, queue);
+ tableIdWriterMap.setRequiredLock(runnable);
+ runnable.run();
+
+ Assertions.assertNull(runnable.getThrowable());
+ Assertions.assertEquals(1, writer.getWriteCount());
+ }
+
+ @Test
+ public void testRunnableDoesNotHoldLockWhileWaitingForRows() throws Exception {
+ BlockingPollQueue queue = new BlockingPollQueue();
+ Map<String, SinkWriter<SeaTunnelRow, ?, ?>> tableIdWriterMap = new HashMap<>();
+ tableIdWriterMap.put("test.table", new RecordingSinkWriter(false));
+ MultiTableWriterRunnable runnable = new MultiTableWriterRunnable(tableIdWriterMap, queue);
+ Thread worker = new Thread(runnable);
+ worker.start();
+
+ Assertions.assertTrue(queue.awaitPollStarted());
+ FutureTask<Boolean> lockProbe =
+ new FutureTask<>(
+ () -> {
+ synchronized (runnable) {
+ return true;
+ }
+ });
+ Thread lockProbeThread = new Thread(lockProbe);
+ lockProbeThread.start();
+
+ try {
+ Assertions.assertTrue(lockProbe.get(200, TimeUnit.MILLISECONDS));
+ } finally {
+ queue.releasePoll();
+ worker.interrupt();
+ worker.join(1000);
+ lockProbeThread.join(1000);
+ }
+ }
+
+ @Test
+ public void testSingleWriterAcceptsNullTableId() throws IOException {
+ Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> sinkWriters = new HashMap<>();
+ Map<SinkIdentifier, SinkWriter.Context> sinkWritersContext = new HashMap<>();
+ RecordingSinkWriter onlyWriter = new RecordingSinkWriter(false);
+ SinkIdentifier sinkIdentifier = SinkIdentifier.of(TablePath.DEFAULT.toString(), 0);
+ sinkWriters.put(sinkIdentifier, onlyWriter);
+ sinkWritersContext.put(sinkIdentifier, new TestSinkWriterContext());
+
+ MultiTableSinkWriter multiTableSinkWriter =
+ new MultiTableSinkWriter(sinkWriters, 1, sinkWritersContext);
+
+ multiTableSinkWriter.write(buildRow(null, 1));
+ Optional<MultiTableCommitInfo> commitInfo = multiTableSinkWriter.prepareCommit(1L);
+
+ Assertions.assertTrue(commitInfo.isPresent());
+ Assertions.assertEquals(1, onlyWriter.getWriteCount());
+ }
+
+ @Test
+ public void testFailedTableMetadataIsSerializable() throws IOException {
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ "test.skipped",
+ MultiTableFailurePhase.SINK_INIT,
+ "console",
+ new RuntimeException("startup failure"));
+ DefaultSerializer<MultiTableFailedTable> serializer = new DefaultSerializer<>();
+
+ byte[] bytes = serializer.serialize(failedTable);
+ MultiTableFailedTable restored = serializer.deserialize(bytes);
+
+ Assertions.assertEquals("test.skipped", restored.getTablePath());
+ Assertions.assertEquals(MultiTableFailurePhase.SINK_INIT, restored.getPhase());
+ Assertions.assertEquals("console", restored.getPluginName());
+ Assertions.assertEquals("RuntimeException", restored.getExceptionClass());
+ Assertions.assertEquals("startup failure", restored.getMessageSummary());
+ Assertions.assertEquals(failedTable.getFirstFailureTime(), restored.getFirstFailureTime());
+ Assertions.assertNull(restored.getCause());
+ }
+
+ private SeaTunnelRow buildRow(String tableId, int value) {
+ SeaTunnelRow row = new SeaTunnelRow(new Object[] {value});
+ row.setTableId(tableId);
+ return row;
+ }
+
static class TestSinkWriter
implements SinkWriter<SeaTunnelRow, TestSinkState, Object>,
SupportMultiTableSinkWriter {
@@ -83,6 +486,154 @@ public class MultiTableSinkWriterTest {
@Override
public void close() throws IOException {}
+
+ @Override
+ public Optional<Integer> primaryKey() {
+ return Optional.of(0);
+ }
+ }
+
+ static class RecordingSinkWriter extends TestSinkWriter {
+ private final boolean failOnWrite;
+ private final boolean interruptAfterWrite;
+ private final AtomicInteger writeCount = new AtomicInteger();
+
+ RecordingSinkWriter(boolean failOnWrite) {
+ this(failOnWrite, false);
+ }
+
+ RecordingSinkWriter(boolean failOnWrite, boolean interruptAfterWrite) {
+ this.failOnWrite = failOnWrite;
+ this.interruptAfterWrite = interruptAfterWrite;
+ }
+
+ @Override
+ public void write(SeaTunnelRow seaTunnelRow) {
+ writeCount.incrementAndGet();
+ if (failOnWrite) {
+ throw new RuntimeException("intentional sink failure");
+ }
+ if (interruptAfterWrite) {
+ Thread.currentThread().interrupt();
+ }
+ }
+
+ int getWriteCount() {
+ return writeCount.get();
+ }
+ }
+
+ static class RetryableWriteSinkWriter extends RecordingSinkWriter {
+ private final int failuresBeforeSuccess;
+
+ RetryableWriteSinkWriter(int failuresBeforeSuccess) {
+ super(false);
+ this.failuresBeforeSuccess = failuresBeforeSuccess;
+ }
+
+ @Override
+ public void write(SeaTunnelRow seaTunnelRow) {
+ super.write(seaTunnelRow);
+ if (getWriteCount() <= failuresBeforeSuccess) {
+ throw new RuntimeException("temporary sink failure");
+ }
+ }
+ }
+
+ static class PrepareCommitRetrySinkWriter extends RecordingSinkWriter {
+ private final int failuresBeforeSuccess;
+ private final AtomicInteger prepareCommitCount = new AtomicInteger();
+
+ PrepareCommitRetrySinkWriter(int failuresBeforeSuccess) {
+ super(false);
+ this.failuresBeforeSuccess = failuresBeforeSuccess;
+ }
+
+ @Override
+ public Optional<TestSinkState> prepareCommit(long checkpointId) throws IOException {
+ int count = prepareCommitCount.incrementAndGet();
+ if (count <= failuresBeforeSuccess) {
+ throw new IOException("temporary prepare commit failure");
+ }
+ return Optional.of(new TestSinkState("retry"));
+ }
+
+ int getPrepareCommitCount() {
+ return prepareCommitCount.get();
+ }
+ }
+
+ static class SnapshotRetrySinkWriter extends RecordingSinkWriter {
+ private final int failuresBeforeSuccess;
+ private final AtomicInteger snapshotAttemptCount = new AtomicInteger();
+ private final AtomicInteger snapshotCount = new AtomicInteger();
+
+ SnapshotRetrySinkWriter(int failuresBeforeSuccess) {
+ super(false);
+ this.failuresBeforeSuccess = failuresBeforeSuccess;
+ }
+
+ @Override
+ public List<Object> snapshotState(long checkpointId) throws IOException {
+ int count = snapshotAttemptCount.incrementAndGet();
+ if (count <= failuresBeforeSuccess) {
+ throw new IOException("temporary snapshot failure");
+ }
+ snapshotCount.incrementAndGet();
+ return Collections.singletonList(new TestSinkState("snapshot"));
+ }
+
+ int getSnapshotAttemptCount() {
+ return snapshotAttemptCount.get();
+ }
+
+ int getSnapshotCount() {
+ return snapshotCount.get();
+ }
+ }
+
+ static class GuardedWriterMap extends HashMap<String, SinkWriter<SeaTunnelRow, ?, ?>> {
+ private static final long serialVersionUID = 1L;
+
+ private Object requiredLock;
+
+ void setRequiredLock(Object requiredLock) {
+ this.requiredLock = requiredLock;
+ }
+
+ @Override
+ public SinkWriter<SeaTunnelRow, ?, ?> get(Object key) {
+ assertLocked();
+ return super.get(key);
+ }
+
+ private void assertLocked() {
+ if (requiredLock != null && !Thread.holdsLock(requiredLock)) {
+ throw new AssertionError("table writer map must be read under runnable lock");
+ }
+ }
+ }
+
+ static class BlockingPollQueue extends LinkedBlockingQueue<SeaTunnelRow> {
+ private static final long serialVersionUID = 1L;
+
+ private final CountDownLatch pollStarted = new CountDownLatch(1);
+ private final CountDownLatch releasePoll = new CountDownLatch(1);
+
+ @Override
+ public SeaTunnelRow poll(long timeout, TimeUnit unit) throws InterruptedException {
+ pollStarted.countDown();
+ releasePoll.await();
+ return null;
+ }
+
+ boolean awaitPollStarted() throws InterruptedException {
+ return pollStarted.await(1, TimeUnit.SECONDS);
+ }
+
+ void releasePoll() {
+ releasePoll.countDown();
+ }
}
static class TestSinkWriterContext implements SinkWriter.Context {
diff --git a/seatunnel-api/src/test/java/org/apache/seatunnel/api/table/catalog/CatalogTableTest.java b/seatunnel-api/src/test/java/org/apache/seatunnel/api/table/catalog/CatalogTableTest.java
index 2003d7add5..e56cc70a02 100644
--- a/seatunnel-api/src/test/java/org/apache/seatunnel/api/table/catalog/CatalogTableTest.java
+++ b/seatunnel-api/src/test/java/org/apache/seatunnel/api/table/catalog/CatalogTableTest.java
@@ -17,8 +17,13 @@
package org.apache.seatunnel.api.table.catalog;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.options.ConnectorCommonOptions;
+import org.apache.seatunnel.api.options.MultiTableCommonOptions;
+import org.apache.seatunnel.api.options.MultiTableFailurePolicy;
import org.apache.seatunnel.api.table.factory.TableSinkFactoryContext;
import org.apache.seatunnel.api.table.factory.TableTransformFactoryContext;
import org.apache.seatunnel.api.table.type.BasicType;
@@ -28,10 +33,12 @@ import org.apache.seatunnel.common.utils.SeaTunnelException;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedHashMap;
+import java.util.List;
import java.util.Map;
public class CatalogTableTest {
@@ -96,6 +103,34 @@ public class CatalogTableTest {
Assertions.assertEquals(result, exception.getParamsValueAs("tableUnsupportedTypes"));
}
+ @Test
+ public void testContinueOtherTablesRecordsDiscoveryFailures() {
+ Catalog catalog =
+ new InMemoryCatalogFactory()
+ .createCatalog("InMemory", ReadonlyConfig.fromMap(new HashMap<>()));
+ Map<String, Object> configMap = new HashMap<>();
+ configMap.put(
+ ConnectorCommonOptions.TABLE_NAMES.key(),
+ Arrays.asList("st.public.table1", "unsupported.public.table1"));
+ configMap.put(
+ MultiTableCommonOptions.MULTI_TABLE_FAILURE_POLICY.key(),
+ MultiTableFailurePolicy.CONTINUE_OTHER_TABLES);
+ List<MultiTableFailedTable> failedTables = new ArrayList<>();
+
+ List<CatalogTable> catalogTables =
+ MultiTableFailureHelper.collectFailedTables(
+ failedTables, () -> catalog.getTables(ReadonlyConfig.fromMap(configMap)));
+
+ Assertions.assertEquals(1, catalogTables.size());
+ Assertions.assertEquals(
+ "st.public.table1", catalogTables.get(0).getTablePath().getFullName());
+ Assertions.assertEquals(1, failedTables.size());
+ MultiTableFailedTable failedTable = failedTables.get(0);
+ Assertions.assertEquals("unsupported.public.table1", failedTable.getTablePath());
+ Assertions.assertEquals(MultiTableFailurePhase.DISCOVERY, failedTable.getPhase());
+ Assertions.assertTrue(failedTable.getMessageSummary().contains("unsupported"));
+ }
+
@Test
public void testCatalogTableWithIllegalFieldNames() {
CatalogTable catalogTable =
diff --git a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/duckdb/DuckDBCatalog.java b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/duckdb/DuckDBCatalog.java
index a3d74ebcfa..a01c0f152d 100644
--- a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/duckdb/DuckDBCatalog.java
+++ b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/duckdb/DuckDBCatalog.java
@@ -135,7 +135,7 @@ public class DuckDBCatalog extends AbstractJdbcCatalog {
if (tableNames != null && !tableNames.isEmpty()) {
Iterator<TablePath> tablePaths =
tableNames.stream().map(TablePath::of).filter(this::tableExists).iterator();
- return buildCatalogTablesWithErrorCheck(tablePaths);
+ return buildCatalogTablesWithErrorCheck(tablePaths, config);
}
// Get the list of table pattern
String tablePatternStr = config.get(ConnectorCommonOptions.TABLE_PATTERN);
@@ -154,7 +154,7 @@ public class DuckDBCatalog extends AbstractJdbcCatalog {
tablePaths.add(tablePath);
}
}
- return buildCatalogTablesWithErrorCheck(tablePaths.iterator());
+ return buildCatalogTablesWithErrorCheck(tablePaths.iterator(), config);
}
protected String getSelectColumnsSql(TablePath tablePath) {
diff --git a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcSourceConfig.java b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcSourceConfig.java
index 093c66d4df..a890abf968 100644
--- a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcSourceConfig.java
+++ b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcSourceConfig.java
@@ -18,6 +18,8 @@
package org.apache.seatunnel.connectors.seatunnel.jdbc.config;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.options.MultiTableCommonOptions;
+import org.apache.seatunnel.api.options.MultiTableFailurePolicy;
import org.apache.seatunnel.connectors.seatunnel.jdbc.source.StringSplitMode;
import org.apache.seatunnel.connectors.seatunnel.jdbc.source.StringSplitStrategy;
@@ -47,6 +49,7 @@ public class JdbcSourceConfig implements Serializable {
private boolean splitSampleShardingAllow;
private boolean decimalTypeNarrowing;
private boolean handleBlobAsString;
+ private MultiTableFailurePolicy multiTableFailurePolicy;
private StringSplitMode stringSplitMode;
@@ -81,6 +84,8 @@ public class JdbcSourceConfig implements Serializable {
builder.decimalTypeNarrowing(config.get(JdbcSourceOptions.DECIMAL_TYPE_NARROWING));
builder.handleBlobAsString(config.get(JdbcSourceOptions.HANDLE_BLOB_AS_STRING));
+ builder.multiTableFailurePolicy(
+ config.get(MultiTableCommonOptions.MULTI_TABLE_FAILURE_POLICY));
config.getOptional(JdbcSourceOptions.WHERE_CONDITION)
.ifPresent(
diff --git a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/JdbcSource.java b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/JdbcSource.java
index 616842bc7f..008d79f697 100644
--- a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/JdbcSource.java
+++ b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/JdbcSource.java
@@ -65,7 +65,8 @@ public class JdbcSource
this.jdbcSourceTables =
JdbcCatalogUtils.getTables(
jdbcSourceConfig.getJdbcConnectionConfig(),
- jdbcSourceConfig.getTableConfigList());
+ jdbcSourceConfig.getTableConfigList(),
+ jdbcSourceConfig.getMultiTableFailurePolicy());
}
@Override
diff --git a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/utils/JdbcCatalogUtils.java b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/utils/JdbcCatalogUtils.java
index ba07f04108..8b979dc2c5 100644
--- a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/utils/JdbcCatalogUtils.java
+++ b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/utils/JdbcCatalogUtils.java
@@ -20,8 +20,12 @@ package org.apache.seatunnel.connectors.seatunnel.jdbc.utils;
import org.apache.seatunnel.shade.com.google.common.base.Strings;
import org.apache.seatunnel.shade.org.apache.commons.lang3.StringUtils;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.options.ConnectorCommonOptions;
+import org.apache.seatunnel.api.options.MultiTableFailurePolicy;
import org.apache.seatunnel.api.table.catalog.Catalog;
import org.apache.seatunnel.api.table.catalog.CatalogTable;
import org.apache.seatunnel.api.table.catalog.Column;
@@ -66,9 +70,12 @@ public class JdbcCatalogUtils {
private static final String DOT_PLACEHOLDER = "__$DOT$__";
public static Map<TablePath, JdbcSourceTable> getTables(
- JdbcConnectionConfig jdbcConnectionConfig, List<JdbcSourceTableConfig> tablesConfig)
+ JdbcConnectionConfig jdbcConnectionConfig,
+ List<JdbcSourceTableConfig> tablesConfig,
+ MultiTableFailurePolicy failurePolicy)
throws SQLException, ClassNotFoundException {
Map<TablePath, JdbcSourceTable> tables = new LinkedHashMap<>();
+ boolean continueOtherTables = failurePolicy != null && failurePolicy.continueOtherTables();
JdbcDialect jdbcDialect =
JdbcDialectLoader.load(
@@ -117,12 +124,29 @@ public class JdbcCatalogUtils {
.equals(
CommonErrorCode
.GET_CATALOG_TABLE_WITH_UNSUPPORTED_TYPE_ERROR)) {
- unsupportedTable.put(
- e.getParams().get("tableName"),
- e.getParamsValueAsMap("fieldWithDataTypes"));
+ if (continueOtherTables) {
+ logSkipFailedTable(
+ e.getParams().get("tableName"),
+ jdbcDialect.dialectName(),
+ e);
+ } else {
+ unsupportedTable.put(
+ e.getParams().get("tableName"),
+ e.getParamsValueAsMap("fieldWithDataTypes"));
+ }
+ } else if (continueOtherTables) {
+ logSkipFailedTable(
+ tableConfig.getTablePath(), jdbcDialect.dialectName(), e);
} else {
throw e;
}
+ } catch (Exception e) {
+ if (continueOtherTables) {
+ logSkipFailedTable(
+ tableConfig.getTablePath(), jdbcDialect.dialectName(), e);
+ } else {
+ throw wrapThrowable(e);
+ }
}
}
if (!unsupportedTable.isEmpty()) {
@@ -143,24 +167,34 @@ public class JdbcCatalogUtils {
try (Connection connection = getConnection(jdbcConnectionConfig, jdbcDialect)) {
log.info("Loading catalog tables for jdbc : {}", jdbcConnectionConfig.getUrl());
for (JdbcSourceTableConfig tableConfig : tablesConfig) {
- CatalogTable catalogTable = getCatalogTable(tableConfig, connection, jdbcDialect);
- TablePath tablePath = catalogTable.getTableId().toTablePath();
- JdbcSourceTable jdbcSourceTable =
- JdbcSourceTable.builder()
- .tablePath(tablePath)
- .query(tableConfig.getQuery())
- .partitionColumn(tableConfig.getPartitionColumn())
- .partitionNumber(tableConfig.getPartitionNumber())
- .partitionStart(tableConfig.getPartitionStart())
- .partitionEnd(tableConfig.getPartitionEnd())
- .useSelectCount(tableConfig.getUseSelectCount())
- .skipAnalyze(tableConfig.getSkipAnalyze())
- .catalogTable(catalogTable)
- .build();
-
- tables.put(tablePath, jdbcSourceTable);
- if (log.isDebugEnabled()) {
- log.debug("Loaded catalog table : {}, {}", tablePath, jdbcSourceTable);
+ try {
+ CatalogTable catalogTable =
+ getCatalogTable(tableConfig, connection, jdbcDialect);
+ TablePath tablePath = catalogTable.getTableId().toTablePath();
+ JdbcSourceTable jdbcSourceTable =
+ JdbcSourceTable.builder()
+ .tablePath(tablePath)
+ .query(tableConfig.getQuery())
+ .partitionColumn(tableConfig.getPartitionColumn())
+ .partitionNumber(tableConfig.getPartitionNumber())
+ .partitionStart(tableConfig.getPartitionStart())
+ .partitionEnd(tableConfig.getPartitionEnd())
+ .useSelectCount(tableConfig.getUseSelectCount())
+ .skipAnalyze(tableConfig.getSkipAnalyze())
+ .catalogTable(catalogTable)
+ .build();
+
+ tables.put(tablePath, jdbcSourceTable);
+ if (log.isDebugEnabled()) {
+ log.debug("Loaded catalog table : {}, {}", tablePath, jdbcSourceTable);
+ }
+ } catch (Exception e) {
+ if (continueOtherTables) {
+ logSkipFailedTable(
+ tableConfig.getTablePath(), jdbcDialect.dialectName(), e);
+ } else {
+ throw wrapThrowable(e);
+ }
}
}
log.info(
@@ -171,6 +205,31 @@ public class JdbcCatalogUtils {
}
}
+ private static void logSkipFailedTable(String tablePath, String pluginName, Throwable error) {
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ tablePath, MultiTableFailurePhase.DISCOVERY, pluginName, error);
+ MultiTableFailureHelper.recordFailedTable(failedTable);
+ log.warn(
+ "Skip failed JDBC source table in discovery: {}",
+ MultiTableFailureHelper.formatFailedTableLine(failedTable),
+ error);
+ }
+
+ private static RuntimeException wrapThrowable(Throwable error)
+ throws SQLException, ClassNotFoundException {
+ if (error instanceof SQLException) {
+ throw (SQLException) error;
+ }
+ if (error instanceof ClassNotFoundException) {
+ throw (ClassNotFoundException) error;
+ }
+ if (error instanceof RuntimeException) {
+ return (RuntimeException) error;
+ }
+ return new RuntimeException(error);
+ }
+
private static CatalogTable getCatalogTable(
JdbcSourceTableConfig tableConfig,
AbstractJdbcCatalog jdbcCatalog,
diff --git a/seatunnel-core/seatunnel-flink-starter/seatunnel-flink-13-starter/src/main/java/org/apache/seatunnel/core/starter/flink/execution/SinkExecuteProcessor.java b/seatunnel-core/seatunnel-flink-starter/seatunnel-flink-13-starter/src/main/java/org/apache/seatunnel/core/starter/flink/execution/SinkExecuteProcessor.java
index 8259302dca..8c31ea3c15 100644
--- a/seatunnel-core/seatunnel-flink-starter/seatunnel-flink-13-starter/src/main/java/org/apache/seatunnel/core/starter/flink/execution/SinkExecuteProcessor.java
+++ b/seatunnel-core/seatunnel-flink-starter/seatunnel-flink-13-starter/src/main/java/org/apache/seatunnel/core/starter/flink/execution/SinkExecuteProcessor.java
@@ -22,6 +22,9 @@ import org.apache.seatunnel.shade.com.typesafe.config.Config;
import org.apache.seatunnel.api.common.JobContext;
import org.apache.seatunnel.api.common.PluginIdentifier;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.options.EnvCommonOptions;
import org.apache.seatunnel.api.sink.SaveModeExecuteWrapper;
@@ -54,6 +57,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.net.URL;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -127,29 +131,70 @@ public class SinkExecuteProcessor
DataStreamTableInfo input = upstreamDataStreams.get(upstreamDataStreams.size() - 1);
Function<PluginIdentifier, SeaTunnelSink> fallbackCreateSink =
sinkPluginDiscovery::createPluginInstance;
+ List<MultiTableFailedTable> skippedTables = new ArrayList<>();
+ boolean createdAnySink = false;
for (int i = 0; i < plugins.size(); i++) {
Optional<? extends Factory> factory = plugins.get(i);
Config sinkConfig = pluginConfigs.get(i);
DataStreamTableInfo stream =
fromSourceTable(sinkConfig, upstreamDataStreams).orElse(input);
Map<TablePath, SeaTunnelSink> sinks = new HashMap<>();
+ List<MultiTableFailedTable> currentSkippedTables = new ArrayList<>();
for (CatalogTable catalogTable : stream.getCatalogTables()) {
- SeaTunnelSink sink =
- FactoryUtil.createAndPrepareSink(
- catalogTable,
- ReadonlyConfig.fromConfig(sinkConfig),
- classLoader,
- sinkConfig.getString(PLUGIN_NAME.key()),
- fallbackCreateSink,
- ((TableSinkFactory) (factory.orElse(null))));
- sink.setJobContext(jobContext);
- handleSaveMode(sink);
- TableIdentifier tableId = catalogTable.getTableId();
- sinks.put(tableId.toTablePath(), sink);
+ SeaTunnelSink sink;
+ try {
+ sink =
+ FactoryUtil.createAndPrepareSink(
+ catalogTable,
+ ReadonlyConfig.fromConfig(sinkConfig),
+ classLoader,
+ sinkConfig.getString(PLUGIN_NAME.key()),
+ fallbackCreateSink,
+ ((TableSinkFactory) (factory.orElse(null))));
+ sink.setJobContext(jobContext);
+ } catch (Exception error) {
+ if (!shouldContinueOtherTables()) {
+ throw wrapThrowable(error);
+ }
+ logSkippedTable(
+ currentSkippedTables,
+ skippedTables,
+ catalogTable,
+ sinkConfig,
+ MultiTableFailurePhase.SINK_INIT,
+ error);
+ continue;
+ }
+ try {
+ handleSaveMode(sink);
+ TableIdentifier tableId = catalogTable.getTableId();
+ sinks.put(tableId.toTablePath(), sink);
+ } catch (Exception error) {
+ if (!shouldContinueOtherTables()) {
+ throw wrapThrowable(error);
+ }
+ logSkippedTable(
+ currentSkippedTables,
+ skippedTables,
+ catalogTable,
+ sinkConfig,
+ MultiTableFailurePhase.SAVE_MODE,
+ error);
+ }
+ }
+ if (sinks.isEmpty()) {
+ continue;
}
SeaTunnelSink sink =
tryGenerateMultiTableSink(
- sinks, ReadonlyConfig.fromConfig(sinkConfig), classLoader);
+ sinks,
+ MultiTableFailureHelper.withFailedTables(
+ MultiTableFailureHelper.mergeOptions(
+ ReadonlyConfig.fromConfig(sinkConfig),
+ ReadonlyConfig.fromConfig(envConfig)),
+ currentSkippedTables),
+ classLoader);
+ createdAnySink = true;
boolean sinkParallelism = sinkConfig.hasPath(EnvCommonOptions.PARALLELISM.key());
boolean envParallelism = envConfig.hasPath(EnvCommonOptions.PARALLELISM.key());
int parallelism =
@@ -180,6 +225,17 @@ public class SinkExecuteProcessor
.name(String.format("%s-Sink", sink.getPluginName()));
dataStreamSink.setParallelism(parallelism);
}
+ if (!createdAnySink && !skippedTables.isEmpty()) {
+ throw new TaskExecuteException(
+ MultiTableFailureHelper.formatFailedTableSummary(
+ "All candidate sink tables were skipped in Flink starter.",
+ skippedTables));
+ }
+ if (createdAnySink && !skippedTables.isEmpty()) {
+ LOGGER.warn(
+ MultiTableFailureHelper.formatFailedTableSummary(
+ "Some sink tables were skipped in Flink starter.", skippedTables));
+ }
// the sink is the last stream
return null;
}
@@ -189,6 +245,9 @@ public class SinkExecuteProcessor
Map<TablePath, SeaTunnelSink> sinks,
ReadonlyConfig sinkConfig,
ClassLoader classLoader) {
+ if (sinks.isEmpty()) {
+ return null;
+ }
if (sinks.values().stream().anyMatch(sink -> !(sink instanceof SupportMultiTableSink))) {
LOGGER.info("Unsupported multi table sink api, rollback to sink template");
// choose the first sink
@@ -211,4 +270,37 @@ public class SinkExecuteProcessor
}
}
}
+
+ private boolean shouldContinueOtherTables() {
+ return MultiTableFailureHelper.shouldContinueOtherTables(
+ ReadonlyConfig.fromConfig(envConfig));
+ }
+
+ private RuntimeException wrapThrowable(Throwable error) {
+ if (error instanceof RuntimeException) {
+ return (RuntimeException) error;
+ }
+ return new RuntimeException(error);
+ }
+
+ private void logSkippedTable(
+ List<MultiTableFailedTable> currentSkippedTables,
+ List<MultiTableFailedTable> skippedTables,
+ CatalogTable catalogTable,
+ Config sinkConfig,
+ MultiTableFailurePhase phase,
+ Throwable error) {
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ catalogTable.getTablePath().getFullName(),
+ phase,
+ sinkConfig.getString(PLUGIN_NAME.key()),
+ error);
+ currentSkippedTables.add(failedTable);
+ skippedTables.add(failedTable);
+ LOGGER.warn(
+ "Skip failed sink table in Flink starter: {}",
+ MultiTableFailureHelper.formatFailedTableLine(failedTable),
+ error);
+ }
}
diff --git a/seatunnel-core/seatunnel-flink-starter/seatunnel-flink-starter-common/src/main/java/org/apache/seatunnel/core/starter/flink/execution/AbstractSinkExecuteProcessor.java b/seatunnel-core/seatunnel-flink-starter/seatunnel-flink-starter-common/src/main/java/org/apache/seatunnel/core/starter/flink/execution/AbstractSinkExecuteProcessor.java
index fbd3607a7a..e0133bcdc0 100644
--- a/seatunnel-core/seatunnel-flink-starter/seatunnel-flink-starter-common/src/main/java/org/apache/seatunnel/core/starter/flink/execution/AbstractSinkExecuteProcessor.java
+++ b/seatunnel-core/seatunnel-flink-starter/seatunnel-flink-starter-common/src/main/java/org/apache/seatunnel/core/starter/flink/execution/AbstractSinkExecuteProcessor.java
@@ -22,6 +22,9 @@ import org.apache.seatunnel.shade.com.typesafe.config.Config;
import org.apache.seatunnel.api.common.JobContext;
import org.apache.seatunnel.api.common.PluginIdentifier;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.options.EnvCommonOptions;
import org.apache.seatunnel.api.sink.SaveModeExecuteWrapper;
@@ -49,6 +52,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.net.URL;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -126,6 +130,8 @@ public abstract class AbstractSinkExecuteProcessor
DataStreamTableInfo input = upstreamDataStreams.get(upstreamDataStreams.size() - 1);
Function<PluginIdentifier, SeaTunnelSink> fallbackCreateSink =
sinkPluginDiscovery::createPluginInstance;
+ List<MultiTableFailedTable> skippedTables = new ArrayList<>();
+ boolean createdAnySink = false;
for (int i = 0; i < plugins.size(); i++) {
Optional<? extends Factory> factory = plugins.get(i);
@@ -133,25 +139,65 @@ public abstract class AbstractSinkExecuteProcessor
DataStreamTableInfo stream =
fromSourceTable(sinkConfig, upstreamDataStreams).orElse(input);
Map<TablePath, SeaTunnelSink> sinks = new HashMap<>();
+ List<MultiTableFailedTable> currentSkippedTables = new ArrayList<>();
for (CatalogTable catalogTable : stream.getCatalogTables()) {
- SeaTunnelSink sink =
- FactoryUtil.createAndPrepareSink(
- catalogTable,
- ReadonlyConfig.fromConfig(sinkConfig),
- classLoader,
- sinkConfig.getString(PLUGIN_NAME.key()),
- fallbackCreateSink,
- ((TableSinkFactory) (factory.orElse(null))));
- sink.setJobContext(jobContext);
- handleSaveMode(sink);
- TableIdentifier tableId = catalogTable.getTableId();
- sinks.put(tableId.toTablePath(), sink);
+ SeaTunnelSink sink;
+ try {
+ sink =
+ FactoryUtil.createAndPrepareSink(
+ catalogTable,
+ ReadonlyConfig.fromConfig(sinkConfig),
+ classLoader,
+ sinkConfig.getString(PLUGIN_NAME.key()),
+ fallbackCreateSink,
+ ((TableSinkFactory) (factory.orElse(null))));
+ sink.setJobContext(jobContext);
+ } catch (Exception error) {
+ if (!shouldContinueOtherTables()) {
+ throw wrapThrowable(error);
+ }
+ logSkippedTable(
+ currentSkippedTables,
+ skippedTables,
+ catalogTable,
+ sinkConfig,
+ MultiTableFailurePhase.SINK_INIT,
+ error);
+ continue;
+ }
+ try {
+ handleSaveMode(sink);
+ TableIdentifier tableId = catalogTable.getTableId();
+ sinks.put(tableId.toTablePath(), sink);
+ } catch (Exception error) {
+ if (!shouldContinueOtherTables()) {
+ throw wrapThrowable(error);
+ }
+ logSkippedTable(
+ currentSkippedTables,
+ skippedTables,
+ catalogTable,
+ sinkConfig,
+ MultiTableFailurePhase.SAVE_MODE,
+ error);
+ }
+ }
+
+ if (sinks.isEmpty()) {
+ continue;
}
SeaTunnelSink sink =
tryGenerateMultiTableSink(
- sinks, ReadonlyConfig.fromConfig(sinkConfig), classLoader);
+ sinks,
+ MultiTableFailureHelper.withFailedTables(
+ MultiTableFailureHelper.mergeOptions(
+ ReadonlyConfig.fromConfig(sinkConfig),
+ ReadonlyConfig.fromConfig(envConfig)),
+ currentSkippedTables),
+ classLoader);
+ createdAnySink = true;
boolean sinkParallelism = sinkConfig.hasPath(EnvCommonOptions.PARALLELISM.key());
boolean envParallelism = envConfig.hasPath(EnvCommonOptions.PARALLELISM.key());
@@ -169,6 +215,17 @@ public abstract class AbstractSinkExecuteProcessor
dataStreamSink.setParallelism(parallelism);
}
}
+ if (!createdAnySink && !skippedTables.isEmpty()) {
+ throw new TaskExecuteException(
+ MultiTableFailureHelper.formatFailedTableSummary(
+ "All candidate sink tables were skipped in Flink starter.",
+ skippedTables));
+ }
+ if (createdAnySink && !skippedTables.isEmpty()) {
+ LOGGER.warn(
+ MultiTableFailureHelper.formatFailedTableSummary(
+ "Some sink tables were skipped in Flink starter.", skippedTables));
+ }
// the sink is the last stream
return null;
}
@@ -182,6 +239,9 @@ public abstract class AbstractSinkExecuteProcessor
Map<TablePath, SeaTunnelSink> sinks,
ReadonlyConfig sinkConfig,
ClassLoader classLoader) {
+ if (sinks.isEmpty()) {
+ return null;
+ }
if (sinks.values().stream().anyMatch(sink -> !(sink instanceof SupportMultiTableSink))) {
LOGGER.info("Unsupported multi table sink api, rollback to sink template");
// choose the first sink
@@ -204,4 +264,37 @@ public abstract class AbstractSinkExecuteProcessor
}
}
}
+
+ protected boolean shouldContinueOtherTables() {
+ return MultiTableFailureHelper.shouldContinueOtherTables(
+ ReadonlyConfig.fromConfig(envConfig));
+ }
+
+ protected RuntimeException wrapThrowable(Throwable error) {
+ if (error instanceof RuntimeException) {
+ return (RuntimeException) error;
+ }
+ return new RuntimeException(error);
+ }
+
+ private void logSkippedTable(
+ List<MultiTableFailedTable> currentSkippedTables,
+ List<MultiTableFailedTable> skippedTables,
+ CatalogTable catalogTable,
+ Config sinkConfig,
+ MultiTableFailurePhase phase,
+ Throwable error) {
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ catalogTable.getTablePath().getFullName(),
+ phase,
+ sinkConfig.getString(PLUGIN_NAME.key()),
+ error);
+ currentSkippedTables.add(failedTable);
+ skippedTables.add(failedTable);
+ LOGGER.warn(
+ "Skip failed sink table in Flink starter: {}",
+ MultiTableFailureHelper.formatFailedTableLine(failedTable),
+ error);
+ }
}
diff --git a/seatunnel-core/seatunnel-spark-starter/seatunnel-spark-2-starter/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SinkExecuteProcessor.java b/seatunnel-core/seatunnel-spark-starter/seatunnel-spark-2-starter/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SinkExecuteProcessor.java
index fd364ada39..241dadce7f 100644
--- a/seatunnel-core/seatunnel-spark-starter/seatunnel-spark-2-starter/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SinkExecuteProcessor.java
+++ b/seatunnel-core/seatunnel-spark-starter/seatunnel-spark-2-starter/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SinkExecuteProcessor.java
@@ -22,13 +22,15 @@ import org.apache.seatunnel.shade.com.typesafe.config.Config;
import org.apache.seatunnel.api.common.JobContext;
import org.apache.seatunnel.api.common.PluginIdentifier;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.options.EnvCommonOptions;
import org.apache.seatunnel.api.sink.SaveModeExecuteWrapper;
import org.apache.seatunnel.api.sink.SaveModeHandler;
import org.apache.seatunnel.api.sink.SeaTunnelSink;
import org.apache.seatunnel.api.sink.SupportSaveMode;
-import org.apache.seatunnel.api.sink.multitablesink.MultiTableSink;
import org.apache.seatunnel.api.table.catalog.CatalogTable;
import org.apache.seatunnel.api.table.catalog.TablePath;
import org.apache.seatunnel.api.table.factory.Factory;
@@ -46,6 +48,8 @@ import org.apache.seatunnel.translation.spark.sink.SparkSinkInjector;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
+import lombok.extern.slf4j.Slf4j;
+
import java.net.URL;
import java.util.ArrayList;
import java.util.HashMap;
@@ -59,6 +63,7 @@ import static org.apache.seatunnel.api.common.SeaTunnelAPIErrorCode.HANDLE_SAVE_
import static org.apache.seatunnel.api.options.ConnectorCommonOptions.PLUGIN_NAME;
import static org.apache.seatunnel.api.table.factory.FactoryUtil.discoverOptionalFactory;
+@Slf4j
public class SinkExecuteProcessor
extends SparkAbstractPluginExecuteProcessor<Optional<? extends Factory>> {
@@ -106,6 +111,8 @@ public class SinkExecuteProcessor
DatasetTableInfo input = upstreamDataStreams.get(upstreamDataStreams.size() - 1);
Function<PluginIdentifier, SeaTunnelSink> fallbackCreateSink =
sinkPluginDiscovery::createPluginInstance;
+ List<MultiTableFailedTable> skippedTables = new ArrayList<>();
+ boolean createdAnySink = false;
for (int i = 0; i < plugins.size(); i++) {
Config sinkConfig = pluginConfigs.get(i);
DatasetTableInfo datasetTableInfo =
@@ -126,26 +133,63 @@ public class SinkExecuteProcessor
}
dataset.sparkSession().read().option(EnvCommonOptions.PARALLELISM.key(), parallelism);
Map<TablePath, SeaTunnelSink> sinks = new HashMap<>();
- datasetTableInfo.getCatalogTables().stream()
- .forEach(
- catalogTable -> {
- SeaTunnelSink<Object, Object, Object, Object> sink =
- FactoryUtil.createAndPrepareSink(
- catalogTable,
- ReadonlyConfig.fromConfig(sinkConfig),
- classLoader,
- sinkConfig.getString(PLUGIN_NAME.key()),
- fallbackCreateSink,
- null);
- sink.setJobContext(jobContext);
- sinks.put(catalogTable.getTableId().toTablePath(), sink);
- });
+ List<MultiTableFailedTable> currentSkippedTables = new ArrayList<>();
+ for (CatalogTable catalogTable : datasetTableInfo.getCatalogTables()) {
+ SeaTunnelSink<Object, Object, Object, Object> sink;
+ try {
+ sink =
+ FactoryUtil.createAndPrepareSink(
+ catalogTable,
+ ReadonlyConfig.fromConfig(sinkConfig),
+ classLoader,
+ sinkConfig.getString(PLUGIN_NAME.key()),
+ fallbackCreateSink,
+ null);
+ sink.setJobContext(jobContext);
+ } catch (Exception error) {
+ if (!shouldContinueOtherTables()) {
+ throw wrapThrowable(error);
+ }
+ logSkippedTable(
+ currentSkippedTables,
+ skippedTables,
+ catalogTable,
+ sinkConfig,
+ MultiTableFailurePhase.SINK_INIT,
+ error);
+ continue;
+ }
+ try {
+ handleSaveMode(sink);
+ sinks.put(catalogTable.getTableId().toTablePath(), sink);
+ } catch (Exception error) {
+ if (!shouldContinueOtherTables()) {
+ throw wrapThrowable(error);
+ }
+ logSkippedTable(
+ currentSkippedTables,
+ skippedTables,
+ catalogTable,
+ sinkConfig,
+ MultiTableFailurePhase.SAVE_MODE,
+ error);
+ }
+ }
+ if (sinks.isEmpty()) {
+ continue;
+ }
SeaTunnelSink sink =
tryGenerateMultiTableSink(
- sinks, ReadonlyConfig.fromConfig(sinkConfig), classLoader);
- // TODO modify checkpoint location
- handleSaveMode(sink);
+ sinks,
+ MultiTableFailureHelper.withFailedTables(
+ MultiTableFailureHelper.mergeOptions(
+ ReadonlyConfig.fromConfig(sinkConfig),
+ ReadonlyConfig.fromConfig(
+ sparkRuntimeEnvironment.getConfig())),
+ currentSkippedTables),
+ classLoader);
+ createdAnySink = true;
String applicationId =
sparkRuntimeEnvironment.getSparkSession().sparkContext().applicationId();
CatalogTable[] catalogTables =
@@ -155,6 +199,17 @@ public class SinkExecuteProcessor
.option("checkpointLocation", "/tmp")
.save();
}
+ if (!createdAnySink && !skippedTables.isEmpty()) {
+ throw new TaskExecuteException(
+ MultiTableFailureHelper.formatFailedTableSummary(
+ "All candidate sink tables were skipped in Spark starter.",
+ skippedTables));
+ }
+ if (createdAnySink && !skippedTables.isEmpty()) {
+ log.warn(
+ MultiTableFailureHelper.formatFailedTableSummary(
+ "Some sink tables were skipped in Spark starter.", skippedTables));
+ }
// the sink is the last stream
return null;
}
@@ -171,11 +226,39 @@ public class SinkExecuteProcessor
throw new SeaTunnelRuntimeException(HANDLE_SAVE_MODE_FAILED, e);
}
}
- } else if (sink instanceof MultiTableSink) {
- Map<TablePath, SeaTunnelSink> sinks = ((MultiTableSink) sink).getSinks();
- for (SeaTunnelSink seaTunnelSink : sinks.values()) {
- handleSaveMode(seaTunnelSink);
- }
}
}
+
+ private boolean shouldContinueOtherTables() {
+ return MultiTableFailureHelper.shouldContinueOtherTables(
+ ReadonlyConfig.fromConfig(sparkRuntimeEnvironment.getConfig()));
+ }
+
+ private RuntimeException wrapThrowable(Throwable error) {
+ if (error instanceof RuntimeException) {
+ return (RuntimeException) error;
+ }
+ return new RuntimeException(error);
+ }
+
+ private void logSkippedTable(
+ List<MultiTableFailedTable> currentSkippedTables,
+ List<MultiTableFailedTable> skippedTables,
+ CatalogTable catalogTable,
+ Config sinkConfig,
+ MultiTableFailurePhase phase,
+ Throwable error) {
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ catalogTable.getTablePath().getFullName(),
+ phase,
+ sinkConfig.getString(PLUGIN_NAME.key()),
+ error);
+ currentSkippedTables.add(failedTable);
+ skippedTables.add(failedTable);
+ log.warn(
+ "Skip failed sink table in Spark starter: {}",
+ MultiTableFailureHelper.formatFailedTableLine(failedTable),
+ error);
+ }
}
diff --git a/seatunnel-core/seatunnel-spark-starter/seatunnel-spark-starter-common/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SinkExecuteProcessor.java b/seatunnel-core/seatunnel-spark-starter/seatunnel-spark-starter-common/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SinkExecuteProcessor.java
index 77b45b6727..8019787605 100644
--- a/seatunnel-core/seatunnel-spark-starter/seatunnel-spark-starter-common/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SinkExecuteProcessor.java
+++ b/seatunnel-core/seatunnel-spark-starter/seatunnel-spark-starter-common/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SinkExecuteProcessor.java
@@ -22,13 +22,15 @@ import org.apache.seatunnel.shade.com.typesafe.config.Config;
import org.apache.seatunnel.api.common.JobContext;
import org.apache.seatunnel.api.common.PluginIdentifier;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.options.EnvCommonOptions;
import org.apache.seatunnel.api.sink.SaveModeExecuteWrapper;
import org.apache.seatunnel.api.sink.SaveModeHandler;
import org.apache.seatunnel.api.sink.SeaTunnelSink;
import org.apache.seatunnel.api.sink.SupportSaveMode;
-import org.apache.seatunnel.api.sink.multitablesink.MultiTableSink;
import org.apache.seatunnel.api.table.catalog.CatalogTable;
import org.apache.seatunnel.api.table.catalog.TablePath;
import org.apache.seatunnel.api.table.factory.Factory;
@@ -110,6 +112,8 @@ public class SinkExecuteProcessor
DatasetTableInfo input = upstreamDataStreams.get(upstreamDataStreams.size() - 1);
Function<PluginIdentifier, SeaTunnelSink> fallbackCreateSink =
sinkPluginDiscovery::createPluginInstance;
+ List<MultiTableFailedTable> skippedTables = new ArrayList<>();
+ boolean createdAnySink = false;
for (int i = 0; i < plugins.size(); i++) {
Config sinkConfig = pluginConfigs.get(i);
DatasetTableInfo datasetTableInfo =
@@ -129,25 +133,62 @@ public class SinkExecuteProcessor
}
dataset.sparkSession().read().option(EnvCommonOptions.PARALLELISM.key(), parallelism);
Map<TablePath, SeaTunnelSink> sinks = new HashMap<>();
- datasetTableInfo.getCatalogTables().stream()
- .forEach(
- catalogTable -> {
- SeaTunnelSink<Object, Object, Object, Object> sink =
- FactoryUtil.createAndPrepareSink(
- catalogTable,
- ReadonlyConfig.fromConfig(sinkConfig),
- classLoader,
- sinkConfig.getString(PLUGIN_NAME.key()),
- fallbackCreateSink,
- null);
- sink.setJobContext(jobContext);
- sinks.put(catalogTable.getTableId().toTablePath(), sink);
- });
+ List<MultiTableFailedTable> currentSkippedTables = new ArrayList<>();
+ for (CatalogTable catalogTable : datasetTableInfo.getCatalogTables()) {
+ SeaTunnelSink<Object, Object, Object, Object> sink;
+ try {
+ sink =
+ FactoryUtil.createAndPrepareSink(
+ catalogTable,
+ ReadonlyConfig.fromConfig(sinkConfig),
+ classLoader,
+ sinkConfig.getString(PLUGIN_NAME.key()),
+ fallbackCreateSink,
+ null);
+ sink.setJobContext(jobContext);
+ } catch (Exception error) {
+ if (!shouldContinueOtherTables()) {
+ throw wrapThrowable(error);
+ }
+ logSkippedTable(
+ currentSkippedTables,
+ skippedTables,
+ catalogTable,
+ sinkConfig,
+ MultiTableFailurePhase.SINK_INIT,
+ error);
+ continue;
+ }
+ try {
+ handleSaveMode(sink);
+ sinks.put(catalogTable.getTableId().toTablePath(), sink);
+ } catch (Exception error) {
+ if (!shouldContinueOtherTables()) {
+ throw wrapThrowable(error);
+ }
+ logSkippedTable(
+ currentSkippedTables,
+ skippedTables,
+ catalogTable,
+ sinkConfig,
+ MultiTableFailurePhase.SAVE_MODE,
+ error);
+ }
+ }
+ if (sinks.isEmpty()) {
+ continue;
+ }
SeaTunnelSink sink =
tryGenerateMultiTableSink(
- sinks, ReadonlyConfig.fromConfig(sinkConfig), classLoader);
- // TODO modify checkpoint location
- handleSaveMode(sink);
+ sinks,
+ MultiTableFailureHelper.withFailedTables(
+ MultiTableFailureHelper.mergeOptions(
+ ReadonlyConfig.fromConfig(sinkConfig),
+ ReadonlyConfig.fromConfig(
+ sparkRuntimeEnvironment.getConfig())),
+ currentSkippedTables),
+ classLoader);
+ createdAnySink = true;
String applicationId =
sparkRuntimeEnvironment.getStreamingContext().sparkContext().applicationId();
CatalogTable[] catalogTables =
@@ -158,6 +199,17 @@ public class SinkExecuteProcessor
.mode(SaveMode.Append)
.save();
}
+ if (!createdAnySink && !skippedTables.isEmpty()) {
+ throw new TaskExecuteException(
+ MultiTableFailureHelper.formatFailedTableSummary(
+ "All candidate sink tables were skipped in Spark starter.",
+ skippedTables));
+ }
+ if (createdAnySink && !skippedTables.isEmpty()) {
+ log.warn(
+ MultiTableFailureHelper.formatFailedTableSummary(
+ "Some sink tables were skipped in Spark starter.", skippedTables));
+ }
// the sink is the last stream
return null;
}
@@ -174,11 +226,39 @@ public class SinkExecuteProcessor
throw new SeaTunnelRuntimeException(HANDLE_SAVE_MODE_FAILED, e);
}
}
- } else if (sink instanceof MultiTableSink) {
- Map<TablePath, SeaTunnelSink> sinks = ((MultiTableSink) sink).getSinks();
- for (SeaTunnelSink seaTunnelSink : sinks.values()) {
- handleSaveMode(seaTunnelSink);
- }
}
}
+
+ private boolean shouldContinueOtherTables() {
+ return MultiTableFailureHelper.shouldContinueOtherTables(
+ ReadonlyConfig.fromConfig(sparkRuntimeEnvironment.getConfig()));
+ }
+
+ private RuntimeException wrapThrowable(Throwable error) {
+ if (error instanceof RuntimeException) {
+ return (RuntimeException) error;
+ }
+ return new RuntimeException(error);
+ }
+
+ private void logSkippedTable(
+ List<MultiTableFailedTable> currentSkippedTables,
+ List<MultiTableFailedTable> skippedTables,
+ CatalogTable catalogTable,
+ Config sinkConfig,
+ MultiTableFailurePhase phase,
+ Throwable error) {
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ catalogTable.getTablePath().getFullName(),
+ phase,
+ sinkConfig.getString(PLUGIN_NAME.key()),
+ error);
+ currentSkippedTables.add(failedTable);
+ skippedTables.add(failedTable);
+ log.warn(
+ "Skip failed sink table in Spark starter: {}",
+ MultiTableFailureHelper.formatFailedTableLine(failedTable),
+ error);
+ }
}
diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-common/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/AbstractJdbcIT.java b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-common/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/AbstractJdbcIT.java
index c92c887ca6..31274185ea 100644
--- a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-common/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/AbstractJdbcIT.java
+++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-common/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/AbstractJdbcIT.java
@@ -22,6 +22,7 @@ import org.apache.seatunnel.shade.com.google.common.io.CharStreams;
import org.apache.seatunnel.shade.org.apache.commons.lang3.StringUtils;
import org.apache.seatunnel.shade.org.apache.commons.lang3.tuple.Pair;
+import org.apache.seatunnel.api.options.MultiTableFailurePolicy;
import org.apache.seatunnel.api.table.catalog.Catalog;
import org.apache.seatunnel.api.table.catalog.CatalogTable;
import org.apache.seatunnel.api.table.catalog.ConstraintKey;
@@ -519,7 +520,8 @@ public abstract class AbstractJdbcIT extends TestSuiteBase implements TestResour
.username(jdbcCase.getUserName())
.password(jdbcCase.getPassword())
.build(),
- tablesConfig);
+ tablesConfig,
+ MultiTableFailurePolicy.FAIL_FAST);
Set<TablePath> tablePaths = tables.keySet();
tablePaths.forEach(
diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlMultipleTablesIT.java b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlMultipleTablesIT.java
index e8dd2b37ae..1844533132 100644
--- a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlMultipleTablesIT.java
+++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlMultipleTablesIT.java
@@ -234,6 +234,40 @@ public class JdbcMysqlMultipleTablesIT extends TestSuiteBase implements TestReso
Assertions.assertAll(asserts);
}
+ @TestTemplate
+ public void testMysqlJdbcContinueOtherTablesWhenSinkSchemaMissing(TestContainer container)
+ throws IOException, InterruptedException, SQLException {
+ clearSinkTables();
+ dropTable(SINK_DATABASE, "table2");
+
+ try {
+ Container.ExecResult execResult =
+ container.executeJob(
+ "/jdbc_mysql_source_and_sink_with_missing_sink_table_continue_other_tables.conf");
+
+ String errorOutput =
+ String.valueOf(execResult.getStderr())
+ + String.valueOf(execResult.getStdout())
+ + container.getServerLogs();
+ Assertions.assertTrue(
+ errorOutput.contains("sink.table2") || errorOutput.contains("table2"),
+ "job output should contain the failed sink table, but was: " + errorOutput);
+ Assertions.assertTrue(
+ errorOutput.contains("phase=save_mode")
+ || errorOutput.contains("save_mode")
+ || errorOutput.contains("The sink table not exist"),
+ "job output should contain the save-mode failure detail, but was: "
+ + errorOutput);
+
+ Assertions.assertIterableEquals(
+ query(String.format("SELECT * FROM %s.%s", SOURCE_DATABASE, "table1")),
+ query(String.format("SELECT * FROM %s.%s", SINK_DATABASE, "table1")));
+ } finally {
+ createTables(SINK_DATABASE, Arrays.asList("table2"));
+ clearSinkTables();
+ }
+ }
+
@AfterAll
@Override
public void tearDown() throws Exception {
@@ -327,6 +361,12 @@ public class JdbcMysqlMultipleTablesIT extends TestSuiteBase implements TestReso
}
}
+ private void dropTable(String database, String table) throws SQLException {
+ try (Statement statement = connection.createStatement()) {
+ statement.execute(String.format("drop table if exists %s.%s", database, table));
+ }
+ }
+
private static Pair<String[], List<SeaTunnelRow>> generateTestDataset() {
String[] fieldNames =
new String[] {
diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink_with_missing_sink_table_continue_other_tables.conf b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink_with_missing_sink_table_continue_other_tables.conf
new file mode 100644
index 0000000000..e794ce3cef
--- /dev/null
+++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink_with_missing_sink_table_continue_other_tables.conf
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+ job.retry.times = 0
+ # Disable restart strategy for this test. Validation/save-mode failures should not be retried,
+ # otherwise healthy tables may be appended repeatedly in Flink e2e containers.
+ execution.restart.strategy = "no"
+ multi_table {
+ failure_policy = "CONTINUE_OTHER_TABLES"
+ }
+}
+
+source {
+ jdbc {
+ url = "jdbc:mysql://mysql-e2e:3306/seatunnel"
+ driver = "com.mysql.cj.jdbc.Driver"
+ connection_check_timeout_sec = 100
+ username = "root"
+ password = "Abc!@#135_seatunnel"
+
+ table_list = [
+ {
+ table_path = "source.table1"
+ },
+ {
+ table_path = "source.table2"
+ query = "select * from source.table2"
+ }
+ ]
+ where_condition = "where c_int >= 0"
+ split.size = 8096
+ split.even-distribution.factor.upper-bound = 100
+ split.even-distribution.factor.lower-bound = 0.05
+ split.sample-sharding.threshold = 1000
+ split.inverse-sampling.rate = 1000
+ }
+}
+
+sink {
+ jdbc {
+ url = "jdbc:mysql://mysql-e2e:3306/seatunnel"
+ driver = "com.mysql.cj.jdbc.Driver"
+ username = "root"
+ password = "Abc!@#135_seatunnel"
+
+ database = "sink"
+ table = "${table_name}"
+ generate_sink_sql = true
+ schema_save_mode = "ERROR_WHEN_SCHEMA_NOT_EXIST"
+ data_save_mode = "APPEND_DATA"
+ }
+}
diff --git a/seatunnel-engine/seatunnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/MultipleTableJobConfigParserTest.java b/seatunnel-engine/seatunnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/MultipleTableJobConfigParserTest.java
index f39a32e337..0d784fc6b4 100644
--- a/seatunnel-engine/seatunnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/MultipleTableJobConfigParserTest.java
+++ b/seatunnel-engine/seatunnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/MultipleTableJobConfigParserTest.java
@@ -18,14 +18,32 @@
package org.apache.seatunnel.engine.client;
import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
import org.apache.seatunnel.shade.org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.seatunnel.api.common.JobContext;
+import org.apache.seatunnel.api.common.PluginIdentifier;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.sink.multitablesink.MultiTableSink;
+import org.apache.seatunnel.api.source.Boundedness;
+import org.apache.seatunnel.api.source.SeaTunnelSource;
+import org.apache.seatunnel.api.source.SourceReader;
+import org.apache.seatunnel.api.source.SourceSplit;
+import org.apache.seatunnel.api.source.SourceSplitEnumerator;
import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.type.BasicType;
import org.apache.seatunnel.common.config.Common;
import org.apache.seatunnel.common.config.DeployMode;
import org.apache.seatunnel.core.starter.utils.ConfigBuilder;
import org.apache.seatunnel.engine.common.config.JobConfig;
+import org.apache.seatunnel.engine.common.exception.JobDefineCheckException;
import org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader;
import org.apache.seatunnel.engine.common.utils.IdGenerator;
import org.apache.seatunnel.engine.core.classloader.ClassLoaderService;
@@ -39,14 +57,18 @@ import org.junit.jupiter.api.Test;
import scala.Tuple2;
import java.io.IOException;
+import java.io.Serializable;
import java.net.URL;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Collection;
+import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Function;
public class MultipleTableJobConfigParserTest {
@@ -241,4 +263,197 @@ public class MultipleTableJobConfigParserTest {
}
Assertions.assertInstanceOf(IllegalArgumentException.class, checkExp);
}
+
+ @Test
+ public void testSkipFailedTableWhenContinueOtherTablesEnabled() throws IOException {
+ Common.setDeployMode(DeployMode.CLIENT);
+ String filePath =
+ ContentFormatUtilTest.getResource("/batch_fake_to_console_multi_table.conf");
+ JobConfig jobConfig = new JobConfig();
+ jobConfig.setJobContext(new JobContext());
+ Config baseConfig = ConfigBuilder.of(Paths.get(filePath));
+ Config config =
+ ConfigFactory.parseString(
+ "env { multi_table { failure_policy = \"CONTINUE_OTHER_TABLES\" } }")
+ .withFallback(baseConfig);
+ MultipleTableJobConfigParser jobConfigParser =
+ new MultipleTableJobConfigParser(config, new IdGenerator(), jobConfig) {
+ @Override
+ public Tuple2<String, List<Tuple2<CatalogTable, Action>>> parseSource(
+ int configIndex, Config sourceConfig, ClassLoader classLoader) {
+ Tuple2<String, List<Tuple2<CatalogTable, Action>>> parsedSource =
+ super.parseSource(configIndex, sourceConfig, classLoader);
+ Action action = parsedSource._2().get(0)._2();
+ return new Tuple2<>(
+ parsedSource._1(),
+ Arrays.asList(
+ new Tuple2<>(mockCatalogTable("table1"), action),
+ new Tuple2<>(mockCatalogTable("table2"), action),
+ new Tuple2<>(mockCatalogTable("table3"), action)));
+ }
+
+ @Override
+ protected Optional<SinkAction<?, ?, ?, ?>> createSinkAction(
+ CatalogTable catalogTable,
+ Set<Action> inputActions,
+ org.apache.seatunnel.api.configuration.ReadonlyConfig readonlyConfig,
+ ClassLoader classLoader,
+ Set<URL> factoryUrls,
+ Set<org.apache.seatunnel.engine.core.job.ConnectorJarIdentifier>
+ connectorJarIdentifiers,
+ String factoryId,
+ int parallelism,
+ int configIndex) {
+ if (catalogTable.getTablePath().getTableName().equals("table2")) {
+ return Optional.empty();
+ }
+ return super.createSinkAction(
+ catalogTable,
+ inputActions,
+ readonlyConfig,
+ classLoader,
+ factoryUrls,
+ connectorJarIdentifiers,
+ factoryId,
+ parallelism,
+ configIndex);
+ }
+ };
+
+ ImmutablePair<List<Action>, Set<URL>> parse = jobConfigParser.parse(null);
+ List<Action> actions = parse.getLeft();
+ Assertions.assertEquals(1, actions.size());
+ Assertions.assertInstanceOf(SinkAction.class, actions.get(0));
+ Assertions.assertInstanceOf(MultiTableSink.class, ((SinkAction) actions.get(0)).getSink());
+ MultiTableSink multiTableSink = (MultiTableSink) ((SinkAction) actions.get(0)).getSink();
+ Assertions.assertEquals(
+ 2, multiTableSink.getSinks().size(), multiTableSink.getSinks().keySet().toString());
+ Assertions.assertFalse(
+ multiTableSink.getSinks().keySet().stream()
+ .anyMatch(tablePath -> tablePath.getTableName().equals("table2")));
+ }
+
+ @Test
+ public void testFailWhenNoSourceTablesRemainAfterDiscovery() throws IOException {
+ Common.setDeployMode(DeployMode.CLIENT);
+ String filePath =
+ ContentFormatUtilTest.getResource("/batch_fake_to_console_multi_table.conf");
+ JobConfig jobConfig = new JobConfig();
+ jobConfig.setJobContext(new JobContext());
+ Config baseConfig = ConfigBuilder.of(Paths.get(filePath));
+ Config config =
+ ConfigFactory.parseString(
+ "env { multi_table { failure_policy = \"CONTINUE_OTHER_TABLES\" } }")
+ .withFallback(baseConfig);
+ MultipleTableJobConfigParser jobConfigParser =
+ new MultipleTableJobConfigParser(config, new IdGenerator(), jobConfig) {
+ @Override
+ public Tuple2<String, List<Tuple2<CatalogTable, Action>>> parseSource(
+ int configIndex, Config sourceConfig, ClassLoader classLoader) {
+ return new Tuple2<>("fake", Collections.emptyList());
+ }
+ };
+
+ JobDefineCheckException exception =
+ Assertions.assertThrows(
+ JobDefineCheckException.class, () -> jobConfigParser.parse(null));
+ Assertions.assertTrue(exception.getMessage().contains("No source tables were available"));
+ }
+
+ @Test
+ public void testSourceDiscoveryFailedTableIsPropagatedToMultiTableSink() throws IOException {
+ Common.setDeployMode(DeployMode.CLIENT);
+ String filePath =
+ ContentFormatUtilTest.getResource("/batch_fake_to_console_multi_table.conf");
+ JobConfig jobConfig = new JobConfig();
+ jobConfig.setJobContext(new JobContext());
+ Config baseConfig = ConfigBuilder.of(Paths.get(filePath));
+ Config config =
+ ConfigFactory.parseString(
+ "env { multi_table { failure_policy = \"CONTINUE_OTHER_TABLES\" } }")
+ .withFallback(baseConfig);
+ MultipleTableJobConfigParser jobConfigParser =
+ new MultipleTableJobConfigParser(config, new IdGenerator(), jobConfig) {
+ @Override
+ protected Tuple2<
+ SeaTunnelSource<Object, SourceSplit, Serializable>,
+ List<CatalogTable>>
+ createAndPrepareSource(
+ int configIndex,
+ ReadonlyConfig readonlyConfig,
+ ClassLoader classLoader,
+ String factoryId,
+ Function<PluginIdentifier, SeaTunnelSource>
+ fallbackCreateSource) {
+ MultiTableFailureHelper.recordFailedTable(
+ MultiTableFailureHelper.buildFailedTable(
+ "test_db.test_schema.table2",
+ MultiTableFailurePhase.DISCOVERY,
+ factoryId,
+ new RuntimeException("metadata read failed")));
+ return new Tuple2<>(
+ new TestSeaTunnelSource(),
+ Collections.singletonList(mockCatalogTable("table1")));
+ }
+ };
+
+ ImmutablePair<List<Action>, Set<URL>> parse = jobConfigParser.parse(null);
+ List<Action> actions = parse.getLeft();
+ Assertions.assertEquals(1, actions.size());
+ Assertions.assertInstanceOf(SinkAction.class, actions.get(0));
+ Assertions.assertInstanceOf(MultiTableSink.class, ((SinkAction) actions.get(0)).getSink());
+ MultiTableSink multiTableSink = (MultiTableSink) ((SinkAction) actions.get(0)).getSink();
+ Assertions.assertEquals(1, multiTableSink.getSinks().size());
+ Assertions.assertEquals(1, multiTableSink.getInitialFailedTables().size());
+ MultiTableFailedTable failedTable = multiTableSink.getInitialFailedTables().get(0);
+ Assertions.assertEquals("test_db.test_schema.table2", failedTable.getTablePath());
+ Assertions.assertEquals(MultiTableFailurePhase.DISCOVERY, failedTable.getPhase());
+ Assertions.assertTrue(failedTable.getMessageSummary().contains("metadata read failed"));
+ }
+
+ private CatalogTable mockCatalogTable(String tableName) {
+ return CatalogTable.of(
+ TableIdentifier.of(
+ "test_catalog", TablePath.of("test_db", "test_schema", tableName)),
+ TableSchema.builder()
+ .column(
+ PhysicalColumn.of(
+ "name", BasicType.STRING_TYPE, 0L, true, null, null))
+ .build(),
+ Collections.emptyMap(),
+ Collections.emptyList(),
+ "");
+ }
+
+ private static class TestSeaTunnelSource
+ implements SeaTunnelSource<Object, SourceSplit, Serializable> {
+
+ @Override
+ public Boundedness getBoundedness() {
+ return Boundedness.BOUNDED;
+ }
+
+ @Override
+ public SourceReader<Object, SourceSplit> createReader(SourceReader.Context readerContext) {
+ return null;
+ }
+
+ @Override
+ public SourceSplitEnumerator<SourceSplit, Serializable> createEnumerator(
+ SourceSplitEnumerator.Context<SourceSplit> enumeratorContext) {
+ return null;
+ }
+
+ @Override
+ public SourceSplitEnumerator<SourceSplit, Serializable> restoreEnumerator(
+ SourceSplitEnumerator.Context<SourceSplit> enumeratorContext,
+ Serializable checkpointState) {
+ return null;
+ }
+
+ @Override
+ public String getPluginName() {
+ return "TestSource";
+ }
+ }
}
diff --git a/seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java b/seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java
index f381349a63..5ee8706c95 100644
--- a/seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java
+++ b/seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java
@@ -27,6 +27,9 @@ import org.apache.seatunnel.shade.org.apache.commons.lang3.StringUtils;
import org.apache.seatunnel.shade.org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.seatunnel.api.common.PluginIdentifier;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.configuration.util.ConfigValidator;
import org.apache.seatunnel.api.metadata.MetadataConfig;
@@ -129,6 +132,8 @@ public class MultipleTableJobConfigParser {
private final boolean isStartWithSavePoint;
private final List<JobPipelineCheckpointData> pipelineCheckpoints;
+ private final List<MultiTableFailedTable> failedTables = new ArrayList<>();
+ private final List<MultiTableFailedTable> sourceFailedTables = new ArrayList<>();
private final MetadataConfig metaDataConfig;
@@ -208,6 +213,8 @@ public class MultipleTableJobConfigParser {
}
public ImmutablePair<List<Action>, Set<URL>> parse(ClassLoaderService classLoaderService) {
+ failedTables.clear();
+ sourceFailedTables.clear();
this.fillJobConfigAndCommonJars();
List<? extends Config> sourceConfigs =
TypesafeConfigUtils.getConfigList(
@@ -256,6 +263,15 @@ public class MultipleTableJobConfigParser {
parseSource(configIndex, sourceConfig, sourceAndTransformClassLoader);
tableWithActionMap.put(tuple2._1(), tuple2._2());
}
+ boolean hasSourceTables =
+ tableWithActionMap.values().stream().anyMatch(actions -> !actions.isEmpty());
+ if (!sourceConfigs.isEmpty()
+ && !hasSourceTables
+ && MultiTableFailureHelper.shouldContinueOtherTables(envOptions)) {
+ throw new JobDefineCheckException(
+ "No source tables were available after discovery. "
+ + "Check source-side failed-table warnings for details.");
+ }
log.info("start generating all transforms.");
parseTransforms(transformConfigs, sourceAndTransformClassLoader, tableWithActionMap);
@@ -268,6 +284,16 @@ public class MultipleTableJobConfigParser {
sinkActions.addAll(
parseSink(configIndex, sinkConfig, sinkClassLoader, tableWithActionMap));
}
+ if (sinkActions.isEmpty() && !failedTables.isEmpty()) {
+ throw new JobDefineCheckException(
+ buildFailureSummary(
+ "All candidate sink tables were skipped during job parsing."));
+ }
+ if (!failedTables.isEmpty()) {
+ log.warn(
+ buildFailureSummary(
+ "Some tables were skipped during multi-table job parsing."));
+ }
Set<URL> factoryUrls = getUsedFactoryUrls(sinkActions);
return new ImmutablePair<>(sinkActions, factoryUrls);
} finally {
@@ -350,7 +376,9 @@ public class MultipleTableJobConfigParser {
public Tuple2<String, List<Tuple2<CatalogTable, Action>>> parseSource(
int configIndex, Config sourceConfig, ClassLoader classLoader) {
- final ReadonlyConfig readonlyConfig = ReadonlyConfig.fromConfig(sourceConfig);
+ final ReadonlyConfig readonlyConfig =
+ MultiTableFailureHelper.withMultiTableFailurePolicy(
+ ReadonlyConfig.fromConfig(sourceConfig), envOptions);
final String factoryId = getFactoryId(readonlyConfig);
final String tableId =
readonlyConfig.getOptional(ConnectorCommonOptions.PLUGIN_OUTPUT).orElse(DEFAULT_ID);
@@ -364,29 +392,19 @@ public class MultipleTableJobConfigParser {
return sourcePluginDiscovery.createPluginInstance(pluginIdentifier);
};
- Tuple2<SeaTunnelSource<Object, SourceSplit, Serializable>, List<CatalogTable>> tuple2;
- if (isStartWithSavePoint && pipelineCheckpoints != null && !pipelineCheckpoints.isEmpty()) {
- ChangeStreamTableSourceCheckpoint checkpoint =
- getSourceCheckpoint(configIndex, factoryId);
- tuple2 =
- FactoryUtil.restoreAndPrepareSource(
- readonlyConfig,
- classLoader,
- factoryId,
- checkpoint,
- fallbackCreateSource,
- null,
- metaDataConfig);
- } else {
- tuple2 =
- FactoryUtil.createAndPrepareSource(
- readonlyConfig,
- classLoader,
- factoryId,
- fallbackCreateSource,
- null,
- metaDataConfig);
- }
+ List<MultiTableFailedTable> discoveryFailedTables = new ArrayList<>();
+ Tuple2<SeaTunnelSource<Object, SourceSplit, Serializable>, List<CatalogTable>> tuple2 =
+ MultiTableFailureHelper.collectFailedTables(
+ discoveryFailedTables,
+ () ->
+ createAndPrepareSource(
+ configIndex,
+ readonlyConfig,
+ classLoader,
+ factoryId,
+ fallbackCreateSource));
+ failedTables.addAll(discoveryFailedTables);
+ sourceFailedTables.addAll(discoveryFailedTables);
Set<URL> factoryUrls = new HashSet<>();
factoryUrls.addAll(getSourcePluginJarPaths(sourceConfig));
@@ -406,6 +424,29 @@ public class MultipleTableJobConfigParser {
return new Tuple2<>(tableId, actions);
}
+ protected Tuple2<SeaTunnelSource<Object, SourceSplit, Serializable>, List<CatalogTable>>
+ createAndPrepareSource(
+ int configIndex,
+ ReadonlyConfig readonlyConfig,
+ ClassLoader classLoader,
+ String factoryId,
+ Function<PluginIdentifier, SeaTunnelSource> fallbackCreateSource) {
+ if (isStartWithSavePoint && pipelineCheckpoints != null && !pipelineCheckpoints.isEmpty()) {
+ ChangeStreamTableSourceCheckpoint checkpoint =
+ getSourceCheckpoint(configIndex, factoryId);
+ return FactoryUtil.restoreAndPrepareSource(
+ readonlyConfig,
+ classLoader,
+ factoryId,
+ checkpoint,
+ fallbackCreateSource,
+ null,
+ metaDataConfig);
+ }
+ return FactoryUtil.createAndPrepareSource(
+ readonlyConfig, classLoader, factoryId, fallbackCreateSource, null, metaDataConfig);
+ }
+
public void parseTransforms(
List<? extends Config> transformConfigs,
ClassLoader classLoader,
@@ -571,6 +612,7 @@ public class MultipleTableJobConfigParser {
Set<URL> jarUrls = new HashSet<>();
jarUrls.addAll(getSinkPluginJarPaths(sinkConfig));
List<SinkAction<?, ?, ?, ?>> sinkActions = new ArrayList<>();
+ int failedTableStartIndex = failedTables.size();
// union
if (inputVertices.size() > 1) {
@@ -581,7 +623,7 @@ public class MultipleTableJobConfigParser {
.collect(Collectors.toCollection(LinkedHashSet::new));
checkProducedTypeEquals(inputActions);
Tuple2<CatalogTable, Action> inputActionSample = inputVertices.get(0).get(0);
- SinkAction<?, ?, ?, ?> sinkAction =
+ Optional<SinkAction<?, ?, ?, ?>> sinkAction =
createSinkAction(
inputActionSample._1(),
inputActions,
@@ -592,14 +634,14 @@ public class MultipleTableJobConfigParser {
factoryId,
inputActionSample._2().getParallelism(),
configIndex);
- sinkActions.add(sinkAction);
+ sinkAction.ifPresent(sinkActions::add);
return sinkActions;
}
// TODO move it into tryGenerateMultiTableSink when we don't support sink template
// sink template
for (Tuple2<CatalogTable, Action> tuple : inputVertices.get(0)) {
- SinkAction<?, ?, ?, ?> sinkAction =
+ Optional<SinkAction<?, ?, ?, ?>> sinkAction =
createSinkAction(
tuple._1(),
Collections.singleton(tuple._2()),
@@ -610,11 +652,16 @@ public class MultipleTableJobConfigParser {
factoryId,
tuple._2().getParallelism(),
configIndex);
- sinkActions.add(sinkAction);
+ sinkAction.ifPresent(sinkActions::add);
}
Optional<SinkAction<?, ?, ?, ?>> multiTableSink =
tryGenerateMultiTableSink(
- sinkActions, readonlyConfig, classLoader, factoryId, configIndex);
+ sinkActions,
+ readonlyConfig,
+ classLoader,
+ factoryId,
+ configIndex,
+ getInitialFailedTablesForSink(failedTableStartIndex));
return multiTableSink
.<List<SinkAction<?, ?, ?, ?>>>map(Collections::singletonList)
.orElse(sinkActions);
@@ -625,7 +672,11 @@ public class MultipleTableJobConfigParser {
ReadonlyConfig options,
ClassLoader classLoader,
String factoryId,
- int configIndex) {
+ int configIndex,
+ List<MultiTableFailedTable> skippedTables) {
+ if (sinkActions.isEmpty()) {
+ return Optional.empty();
+ }
if (sinkActions.stream()
.anyMatch(action -> !(action.getSink() instanceof SupportMultiTableSink))) {
log.info("Unsupported multi table sink api, rollback to sink template");
@@ -643,7 +694,12 @@ public class MultipleTableJobConfigParser {
sinks.put(tablePath, sink);
});
SeaTunnelSink<?, ?, ?, ?> sink =
- FactoryUtil.createMultiTableSink(sinks, options, classLoader);
+ FactoryUtil.createMultiTableSink(
+ sinks,
+ MultiTableFailureHelper.withFailedTables(
+ MultiTableFailureHelper.mergeOptions(options, envOptions),
+ skippedTables),
+ classLoader);
String actionName =
JobConfigParser.createSinkActionName(configIndex, factoryId, "MultiTableSink");
SinkAction<?, ?, ?, ?> multiTableAction =
@@ -658,7 +714,19 @@ public class MultipleTableJobConfigParser {
return Optional.of(multiTableAction);
}
- private SinkAction<?, ?, ?, ?> createSinkAction(
+ private List<MultiTableFailedTable> getInitialFailedTablesForSink(int failedTableStartIndex) {
+ Map<String, MultiTableFailedTable> initialFailedTables = new LinkedHashMap<>();
+ sourceFailedTables.forEach(
+ failedTable -> initialFailedTables.put(failedTable.getTablePath(), failedTable));
+ failedTables
+ .subList(failedTableStartIndex, failedTables.size())
+ .forEach(
+ failedTable ->
+ initialFailedTables.put(failedTable.getTablePath(), failedTable));
+ return new ArrayList<>(initialFailedTables.values());
+ }
+
+ protected Optional<SinkAction<?, ?, ?, ?>> createSinkAction(
CatalogTable catalogTable,
Set<Action> inputActions,
ReadonlyConfig readonlyConfig,
@@ -676,14 +744,20 @@ public class MultipleTableJobConfigParser {
return sinkPluginDiscovery.createPluginInstance(pluginIdentifier);
};
- SeaTunnelSink<?, ?, ?, ?> sink =
- FactoryUtil.createAndPrepareSink(
- catalogTable,
- readonlyConfig,
- classLoader,
- factoryId,
- fallbackCreateSink,
- null);
+ SeaTunnelSink<?, ?, ?, ?> sink;
+ try {
+ sink =
+ FactoryUtil.createAndPrepareSink(
+ catalogTable,
+ readonlyConfig,
+ classLoader,
+ factoryId,
+ fallbackCreateSink,
+ null);
+ } catch (Exception error) {
+ return handleCreateSinkFailure(
+ catalogTable, factoryId, MultiTableFailurePhase.SINK_INIT, error);
+ }
sink.setJobContext(jobConfig.getJobContext());
SinkConfig actionConfig = new SinkConfig(catalogTable.getTableId().toTablePath());
long id = idGenerator.getNextId();
@@ -699,13 +773,18 @@ public class MultipleTableJobConfigParser {
factoryUrls,
connectorJarIdentifiers,
actionConfig);
- if (!isStartWithSavePoint) {
- handleSaveMode(sink);
- } else {
- handleSchemaSaveModeWithRestore(sink);
+ try {
+ if (!isStartWithSavePoint) {
+ handleSaveMode(sink);
+ } else {
+ handleSchemaSaveModeWithRestore(sink);
+ }
+ } catch (Exception error) {
+ return handleCreateSinkFailure(
+ catalogTable, factoryId, MultiTableFailurePhase.SAVE_MODE, error);
}
sinkAction.setParallelism(parallelism);
- return sinkAction;
+ return Optional.of(sinkAction);
}
public void handleSaveMode(SeaTunnelSink<?, ?, ?, ?> sink) {
@@ -748,6 +827,36 @@ public class MultipleTableJobConfigParser {
}
}
+ private Optional<SinkAction<?, ?, ?, ?>> handleCreateSinkFailure(
+ CatalogTable catalogTable,
+ String factoryId,
+ MultiTableFailurePhase phase,
+ Throwable error) {
+ if (!MultiTableFailureHelper.shouldContinueOtherTables(envOptions)) {
+ throw wrapThrowable(error);
+ }
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ catalogTable.getTablePath().getFullName(), phase, factoryId, error);
+ failedTables.add(failedTable);
+ log.warn(
+ "Skip failed sink table during parsing: {}",
+ MultiTableFailureHelper.formatFailedTableLine(failedTable),
+ error);
+ return Optional.empty();
+ }
+
+ private String buildFailureSummary(String title) {
+ return MultiTableFailureHelper.formatFailedTableSummary(title, failedTables);
+ }
+
+ private RuntimeException wrapThrowable(Throwable error) {
+ if (error instanceof RuntimeException) {
+ return (RuntimeException) error;
+ }
+ return new RuntimeException(error);
+ }
+
private List<URL> getSourcePluginJarPaths(Config sourceConfig) {
SeaTunnelSourcePluginDiscovery sourcePluginDiscovery = new SeaTunnelSourcePluginDiscovery();
PluginIdentifier pluginIdentifier =
diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java
index 7beb6c0fec..255bcbb112 100644
--- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java
+++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/SubPlan.java
@@ -17,6 +17,7 @@
package org.apache.seatunnel.engine.server.dag.physical;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
import org.apache.seatunnel.api.options.EnvCommonOptions;
import org.apache.seatunnel.common.utils.ExceptionUtils;
import org.apache.seatunnel.common.utils.RetryUtils;
@@ -289,7 +290,9 @@ public class SubPlan {
}
private boolean checkNeedRestore(PipelineStatus pipelineStatus) {
- return canRestorePipeline() && !PipelineStatus.FINISHED.equals(pipelineStatus);
+ return canRestorePipeline()
+ && !PipelineStatus.FINISHED.equals(pipelineStatus)
+ && !MultiTableFailureHelper.isIsolatedFailure(errorByPhysicalVertex.get());
}
/** only call when the pipeline will never restart */
diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java
index 8dad23b736..976791f503 100644
--- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java
+++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java
@@ -21,6 +21,9 @@ import org.apache.seatunnel.shade.com.google.common.annotations.VisibleForTestin
import org.apache.seatunnel.api.common.metrics.JobMetrics;
import org.apache.seatunnel.api.common.metrics.RawJobMetrics;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailedTable;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailureHelper;
+import org.apache.seatunnel.api.common.multitable.MultiTableFailurePhase;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.options.EnvCommonOptions;
import org.apache.seatunnel.api.sink.SaveModeExecuteLocation;
@@ -613,9 +616,48 @@ public class JobMaster {
}
}
} else if (sink instanceof MultiTableSink) {
- Map<TablePath, SeaTunnelSink> sinks = ((MultiTableSink) sink).getSinks();
- for (SeaTunnelSink seaTunnelSink : sinks.values()) {
- handleSaveMode(seaTunnelSink, isStartWithSavePoint);
+ MultiTableSink multiTableSink = (MultiTableSink) sink;
+ Map<TablePath, SeaTunnelSink> sinks = multiTableSink.getSinks();
+ if (!multiTableSink.getFailurePolicy().continueOtherTables()) {
+ for (SeaTunnelSink seaTunnelSink : sinks.values()) {
+ handleSaveMode(seaTunnelSink, isStartWithSavePoint);
+ }
+ return;
+ }
+
+ List<MultiTableFailedTable> failedTables = new ArrayList<>();
+ for (Map.Entry<TablePath, SeaTunnelSink> entry : new ArrayList<>(sinks.entrySet())) {
+ try {
+ handleSaveMode(entry.getValue(), isStartWithSavePoint);
+ } catch (RuntimeException error) {
+ MultiTableFailedTable failedTable =
+ MultiTableFailureHelper.buildFailedTable(
+ entry.getKey().getFullName(),
+ MultiTableFailurePhase.SAVE_MODE,
+ entry.getValue().getPluginName(),
+ error);
+ failedTables.add(failedTable);
+ LOGGER.warning(
+ "Skip failed sink table during cluster save mode: "
+ + MultiTableFailureHelper.formatFailedTableLine(failedTable),
+ error);
+ }
+ }
+ if (failedTables.isEmpty()) {
+ return;
+ }
+
+ failedTables.forEach(
+ failedTable ->
+ multiTableSink.removeSink(TablePath.of(failedTable.getTablePath())));
+ multiTableSink.registerInitialFailedTables(failedTables);
+ if (multiTableSink.getSinks().isEmpty()) {
+ throw new SeaTunnelRuntimeException(
+ HANDLE_SAVE_MODE_FAILED,
+ new IllegalStateException(
+ MultiTableFailureHelper.formatFailedTableSummary(
+ "All candidate sink tables were skipped during cluster save mode.",
+ failedTables)));
}
}
}