You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by gi...@apache.org on 2026/08/31 13:30:30 UTC
(seatunnel) 01/01: [Feature][Connector-V2] BigQuery Sink support for Multi-Table and UniverseDomain (#11649)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-11649-1d18735bd38af023a33ae5ff77a738fcac55a549
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit f0046a0022028f0b511b2920a8a6cb7ded7d0d8b
Author: Merlin-S3NS <me...@s3ns.io>
AuthorDate: Mon Aug 31 13:30:02 2026 +0000
[Feature][Connector-V2] BigQuery Sink support for Multi-Table and UniverseDomain (#11649)
---
docs/en/connectors/sink/BigQuery.md | 35 +-
.../introduction/concepts/incompatible-changes.md | 6 +
docs/zh/connectors/sink/BigQuery.md | 35 +-
seatunnel-connectors-v2/connector-bigquery/pom.xml | 8 +-
.../bigquery/catalog/BigQueryCatalog.java | 517 +++++++++++++++++++++
.../bigquery/client/BigQueryClientFactory.java | 25 +-
.../bigquery/option/BigQuerySinkOptions.java | 27 ++
.../bigquery/sink/AbstractBigQuerySinkWriter.java | 34 +-
.../sink/BigQueryMultiTableResourceManager.java | 67 +++
.../bigquery/sink/BigQuerySaveModeHandler.java | 164 +++++++
.../connectors/bigquery/sink/BigQuerySink.java | 74 +--
.../bigquery/sink/BigQuerySinkBatchWriter.java | 41 +-
.../bigquery/sink/BigQuerySinkFactory.java | 1 +
.../bigquery/sink/BigQuerySinkStreamWriter.java | 19 +-
.../sink/BigQueryCatalogAndSaveModeTest.java | 442 ++++++++++++++++++
.../bigquery/sink/BigQuerySinkBatchWriterTest.java | 21 +-
.../bigquery/sink/BigQuerySinkFactoryTest.java | 22 +
17 files changed, 1463 insertions(+), 75 deletions(-)
diff --git a/docs/en/connectors/sink/BigQuery.md b/docs/en/connectors/sink/BigQuery.md
index 801d05cff5..099c168c48 100644
--- a/docs/en/connectors/sink/BigQuery.md
+++ b/docs/en/connectors/sink/BigQuery.md
@@ -14,7 +14,7 @@ import ChangeLog from '../changelog/connector-bigquery.md';
- [x] [exactly-once](../../introduction/concepts/connector-v2-features.md) for batch mode only
- [x] [cdc](../../introduction/concepts/connector-v2-features.md)
-- [ ] [support multiple table write](../../introduction/concepts/connector-v2-features.md)
+- [x] [support multiple table write](../../introduction/concepts/connector-v2-features.md)
- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
## Description
@@ -41,7 +41,11 @@ Sink connector for Google Cloud BigQuery using the Storage Write API for high-pe
| sequence_number_column | string | No | - | Column name used as sequence number for CDC deduplication. Only applicable when `write_mode` is `streaming` |
| batch_size | int | No | 1000 | Number of rows to batch before sending to BigQuery |
| emulator_host | string | No | - | BigQuery emulator host, such as `localhost:9050`. This option is intended for tests only. |
-| multi_table_sink_replica | int | No | - | Sink common option. It controls sink replica count in multi-table runtime, but this connector still writes to the single configured BigQuery table. |
+| universe_domain | string | No | - | The Google Cloud Universe Domain, such as `s3nsapis.fr` for S3NS sovereign cloud. |
+| schema_save_mode | enum | No | CREATE_SCHEMA_WHEN_NOT_EXIST | Schema save mode. See below. |
+| data_save_mode | enum | No | APPEND_DATA | Data save mode. See below. |
+| custom_sql | string | No | - | Custom SQL to execute when `data_save_mode` is `CUSTOM_PROCESSING`. |
+| multi_table_sink_replica | int | No | - | Sink common option. It controls sink replica count in multi-table runtime. |
| common-options | | No | - | Sink common options. See [Sink Common Options](../common-options/sink-common-options.md). |
### Authentication Options
@@ -54,10 +58,31 @@ For production BigQuery jobs, provide **one** of the following authentication me
### Table Options
-The target BigQuery table must already exist.
-The connector reads the existing table schema during writer initialization and does not create the table automatically.
+The target BigQuery table can be created automatically using SeaTunnel's SaveMode.
+By configuring `schema_save_mode` to `CREATE_SCHEMA_WHEN_NOT_EXIST` (default) or `RECREATE_SCHEMA`, the connector can automatically create the BigQuery dataset and table based on the upstream schema information.
-The connector writes to one configured table: `project_id.dataset_id.table_id`. It does not create a different BigQuery table for each upstream table. For multi-table pipelines, configure separate sink entries or route data before the BigQuery sink.
+The connector writes to target tables determined by `project_id.dataset_id.table_id`.
+In multi-table pipelines, you can configure `table_id` to include `${table_name}` (e.g., `table_id = "${table_name}"` or `table_id = "prefix_${table_name}"`) to dynamically route data to different BigQuery tables. Under this multi-table setting, the connector automatically creates separate target tables as needed.
+
+### schema_save_mode [Enum]
+
+Before the synchronization task starts, controls how the target table schema is handled.
+- `RECREATE_SCHEMA` : Drop the target table if it exists, and then recreate it.
+- `CREATE_SCHEMA_WHEN_NOT_EXIST` : Create the target table if it does not exist, or skip creation if it exists.
+- `ERROR_WHEN_SCHEMA_NOT_EXIST` : Throw an error if the target table does not exist.
+- `IGNORE` : Ignore schema handling and do not perform any schema-related checks or DDL actions.
+
+### data_save_mode [Enum]
+
+Before the synchronization task starts, controls how existing data on the target side is handled.
+- `DROP_DATA` : Delete existing data in the target table.
+- `APPEND_DATA` : Keep the target table's existing structure and append new data.
+- `CUSTOM_PROCESSING` : Perform user-defined processing. This requires configuring `custom_sql`.
+- `ERROR_WHEN_DATA_EXISTS` : Throw an error if the target table already contains data.
+
+### custom_sql [String]
+
+When `data_save_mode` is set to `CUSTOM_PROCESSING`, the SQL statement specified here will be executed before row writing begins.
### Write Modes
diff --git a/docs/en/introduction/concepts/incompatible-changes.md b/docs/en/introduction/concepts/incompatible-changes.md
index b64f17b2b6..6e2e76ae90 100644
--- a/docs/en/introduction/concepts/incompatible-changes.md
+++ b/docs/en/introduction/concepts/incompatible-changes.md
@@ -111,6 +111,12 @@ You need to check this document before you upgrade to related version.
### Connector Changes
+- **Breaking Change: BigQuery Sink Connector — default schema save mode introduces automatic table creation**
+ - **Affected component**: `seatunnel-connectors-v2/connector-bigquery`
+ - **Description**: The BigQuery sink connector (`connector-bigquery`) now implements `SupportSaveMode` with support for `schema_save_mode` and `data_save_mode`. The default `schema_save_mode` is set to `CREATE_SCHEMA_WHEN_NOT_EXIST`.
+ - **Impact**: Upgrading existing pipelines targeting a non-existent table will now automatically create the table in BigQuery with the source schema instead of failing fast at the BigQuery API layer.
+ - **Migration Guide**: To preserve the legacy fail-fast behavior, explicitly configure `schema_save_mode = "ERROR_WHEN_SCHEMA_NOT_EXIST"` in your BigQuery sink configuration.
+
- **Breaking Change: ORC file sink preserves case of nested struct field names**
- **Affected component**: `seatunnel-connectors-v2/connector-file/connector-file-base` (used by all File/HDFS/S3/OSS ORC sinks that share `OrcWriteStrategy`)
- **Description**: Previously, `OrcWriteStrategy.buildFieldWithRowType(...)` forced every nested `ROW` (struct) field name to lowercase when building the ORC schema, so a nested field declared as `MD5` was persisted as `md5` in the file footer. Downstream consumers that read the column by its declared original-case name received null/missing values. The `.toLowerCase()` call has been removed from the recursive nested-field branch, so nested struct field names are now written verbatim i [...]
diff --git a/docs/zh/connectors/sink/BigQuery.md b/docs/zh/connectors/sink/BigQuery.md
index 63e248e2e0..61bb7b87b6 100644
--- a/docs/zh/connectors/sink/BigQuery.md
+++ b/docs/zh/connectors/sink/BigQuery.md
@@ -14,7 +14,7 @@ import ChangeLog from '../changelog/connector-bigquery.md';
- [x] [精确一次](../../introduction/concepts/connector-v2-features.md) 仅适用于 batch 模式
- [x] [CDC](../../introduction/concepts/connector-v2-features.md)
-- [ ] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
@@ -40,7 +40,11 @@ import ChangeLog from '../changelog/connector-bigquery.md';
| sequence_number_column | string | 否 | - | 用于 CDC 去重的序列号列名。仅在 `write_mode` 为 `streaming` 时适用 |
| batch_size | int | 否 | 1000 | 发送到 BigQuery 之前批量处理的行数 |
| emulator_host | string | 否 | - | BigQuery emulator 地址,例如 `localhost:9050`。该参数仅用于测试。 |
-| multi_table_sink_replica | int | 否 | - | Sink 通用参数,用于控制多表运行时每张表的 sink 副本数;但该连接器仍只写入配置中的单个 BigQuery 表。 |
+| universe_domain | string | 否 | - | Google Cloud 宇宙域/环境域名,例如主权云 S3NS 环境配置为 `s3nsapis.fr`。 |
+| schema_save_mode | enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | Schema 保存模式。详见下文。 |
+| data_save_mode | enum | 否 | APPEND_DATA | Data 保存模式。详见下文。 |
+| custom_sql | string | 否 | - | 当 `data_save_mode` 选择 `CUSTOM_PROCESSING` 时,需要填写的自定义 SQL 语句。 |
+| multi_table_sink_replica | int | 否 | - | Sink 通用参数,用于控制多表运行时每张表的 sink 副本数。 |
| common-options | | 否 | - | Sink 通用参数,详见 [Sink Common Options](../common-options/sink-common-options.md)。 |
### 认证参数
@@ -53,10 +57,31 @@ import ChangeLog from '../changelog/connector-bigquery.md';
### 表选项
-目标 BigQuery 表必须已经存在。
-连接器会在 writer 初始化时读取已有的表 schema,并且不会自动创建 BigQuery 表。
+目标 BigQuery 表可以通过 SeaTunnel SaveMode 自动创建。
+通过将 `schema_save_mode` 配置为 `CREATE_SCHEMA_WHEN_NOT_EXIST`(默认值)或 `RECREATE_SCHEMA`,连接器在初始化时可以基于上游 schema 信息自动创建 BigQuery 数据集和数据表。
-该连接器会写入一个固定的目标表:`project_id.dataset_id.table_id`。它不会按上游表自动创建或切换 BigQuery 目标表。如果任务里有多张表,请配置多个 BigQuery sink,或者在写入 BigQuery 前先完成表路由。
+连接器写入的目标表由 `project_id.dataset_id.table_id` 决定。
+在多表同步场景下,您可以将 `table_id` 配置为包含 `${table_name}` 的表达式(例如 `table_id = "${table_name}"` 或 `table_id = "prefix_${table_name}"`),从而将数据动态路由到不同的 BigQuery 表中。在这种多表设置下,连接器将根据上游表信息自动在 BigQuery 中创建相应的目标表。
+
+### schema_save_mode [Enum]
+
+在同步任务启动之前,控制如何处理目标表的结构。
+- `RECREATE_SCHEMA` :如果目标表存在,则先删除该表然后重新创建;如果不存在则直接创建。
+- `CREATE_SCHEMA_WHEN_NOT_EXIST` :如果目标表不存在则创建它;如果已存在则跳过创建。
+- `ERROR_WHEN_SCHEMA_NOT_EXIST` :如果目标表不存在,则抛出异常并报错。
+- `IGNORE` :忽略目标表结构的处理,不执行任何与结构相关的检查或 DDL 动作。
+
+### data_save_mode [Enum]
+
+在同步任务启动之前,控制如何处理目标表中的已有数据。
+- `DROP_DATA` :删除目标表中的已有数据。
+- `APPEND_DATA` :保留目标表中的已有数据,并将新数据追加写入。
+- `CUSTOM_PROCESSING` :执行用户自定义的处理。此选项需要配合 `custom_sql` 参数。
+- `ERROR_WHEN_DATA_EXISTS` :如果目标表已包含数据,则抛出异常并报错。
+
+### custom_sql [String]
+
+当 `data_save_mode` 被设置为 `CUSTOM_PROCESSING` 时,此参数中填写的自定义 SQL 语句将在数据开始写入前被执行。
### 写入模式
diff --git a/seatunnel-connectors-v2/connector-bigquery/pom.xml b/seatunnel-connectors-v2/connector-bigquery/pom.xml
index ed3ad5dac4..d31cc84e84 100644
--- a/seatunnel-connectors-v2/connector-bigquery/pom.xml
+++ b/seatunnel-connectors-v2/connector-bigquery/pom.xml
@@ -103,10 +103,6 @@
<pattern>com.google.gson</pattern>
<shadedPattern>${seatunnel.shade.package}.com.google.gson</shadedPattern>
</relocation>
- <relocation>
- <pattern>com.google.protobuf</pattern>
- <shadedPattern>${seatunnel.shade.package}.com.google.protobuf</shadedPattern>
- </relocation>
<relocation>
<pattern>io.grpc</pattern>
<shadedPattern>${seatunnel.shade.package}.io.grpc</shadedPattern>
@@ -115,6 +111,10 @@
<pattern>org.apache.arrow</pattern>
<shadedPattern>${seatunnel.shade.package}.org.apache.arrow</shadedPattern>
</relocation>
+ <relocation>
+ <pattern>io.netty</pattern>
+ <shadedPattern>${seatunnel.shade.package}.io.netty</shadedPattern>
+ </relocation>
</relocations>
<filters>
<filter>
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/catalog/BigQueryCatalog.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/catalog/BigQueryCatalog.java
new file mode 100644
index 0000000000..e31bf00571
--- /dev/null
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/catalog/BigQueryCatalog.java
@@ -0,0 +1,517 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.bigquery.catalog;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.table.catalog.Catalog;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.catalog.exception.CatalogException;
+import org.apache.seatunnel.api.table.catalog.exception.DatabaseAlreadyExistException;
+import org.apache.seatunnel.api.table.catalog.exception.DatabaseNotExistException;
+import org.apache.seatunnel.api.table.catalog.exception.TableAlreadyExistException;
+import org.apache.seatunnel.api.table.catalog.exception.TableNotExistException;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.PrimitiveByteArrayType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.connectors.bigquery.client.BigQueryClientFactory;
+import org.apache.seatunnel.connectors.bigquery.option.BigQuerySinkOptions;
+import org.apache.seatunnel.connectors.bigquery.sink.BigQuerySinkBatchWriter;
+
+import com.google.api.gax.paging.Page;
+import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.Dataset;
+import com.google.cloud.bigquery.DatasetInfo;
+import com.google.cloud.bigquery.Field;
+import com.google.cloud.bigquery.FieldList;
+import com.google.cloud.bigquery.QueryJobConfiguration;
+import com.google.cloud.bigquery.Schema;
+import com.google.cloud.bigquery.StandardSQLTypeName;
+import com.google.cloud.bigquery.StandardTableDefinition;
+import com.google.cloud.bigquery.Table;
+import com.google.cloud.bigquery.TableDefinition;
+import com.google.cloud.bigquery.TableId;
+import com.google.cloud.bigquery.TableInfo;
+import lombok.extern.slf4j.Slf4j;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.seatunnel.connectors.bigquery.sink.writer.BigQueryStreamWriter.CHANGE_TYPE;
+import static org.apache.seatunnel.connectors.bigquery.sink.writer.BigQueryStreamWriter.SEQUENCE_NUM;
+
+/**
+ * Catalog implementation for Google Cloud BigQuery. This class provides standard catalog operations
+ * such as listing databases/datasets, creating and deleting tables, checking table existence, and
+ * reading metadata.
+ */
+@Slf4j
+public class BigQueryCatalog implements Catalog {
+
+ private final String catalogName;
+ private final ReadonlyConfig config;
+ private BigQuery bigquery;
+
+ /**
+ * Constructs a new BigQueryCatalog.
+ *
+ * @param catalogName the name of this catalog
+ * @param config the readonly configuration options containing BigQuery connection info
+ */
+ public BigQueryCatalog(String catalogName, ReadonlyConfig config) {
+ this.catalogName = catalogName;
+ this.config = config;
+ }
+
+ /**
+ * Opens the catalog and initializes the BigQuery client.
+ *
+ * @throws CatalogException if the BigQuery client fails to initialize
+ */
+ @Override
+ public void open() throws CatalogException {
+ try {
+ this.bigquery = BigQueryClientFactory.getBigQuery(config);
+ log.info("BigQueryCatalog '{}' opened successfully.", catalogName);
+ } catch (Exception e) {
+ throw new CatalogException("Failed to open BigQueryCatalog", e);
+ }
+ }
+
+ /**
+ * Closes the catalog.
+ *
+ * @throws CatalogException if any resources fail to close
+ */
+ @Override
+ public void close() throws CatalogException {
+ // BigQuery service client doesn't hold open TCP sockets directly; it's a stateless HTTP
+ // wrapper.
+ log.info("BigQueryCatalog '{}' closed successfully.", catalogName);
+ }
+
+ /**
+ * Returns the catalog name.
+ *
+ * @return the name of this catalog
+ */
+ @Override
+ public String name() {
+ return catalogName;
+ }
+
+ private String getDatasetName(TablePath tablePath) {
+ String db = tablePath.getDatabaseName();
+ if (db == null || db.trim().isEmpty() || "default".equalsIgnoreCase(db)) {
+ return config.get(BigQuerySinkOptions.DATASET_ID);
+ }
+ return db;
+ }
+
+ /**
+ * Returns the default database (dataset ID) configured for the BigQuery connector.
+ *
+ * @return the configured default dataset ID
+ * @throws CatalogException if default dataset info cannot be retrieved
+ */
+ @Override
+ public String getDefaultDatabase() throws CatalogException {
+ return config.get(BigQuerySinkOptions.DATASET_ID);
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ if (databaseName == null || databaseName.trim().isEmpty()) {
+ databaseName = config.get(BigQuerySinkOptions.DATASET_ID);
+ }
+ return bigquery.getDataset(databaseName) != null;
+ }
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ List<String> databases = new ArrayList<>();
+ try {
+ Page<Dataset> datasets = bigquery.listDatasets();
+ for (Dataset dataset : datasets.iterateAll()) {
+ databases.add(dataset.getDatasetId().getDataset());
+ }
+ } catch (Exception e) {
+ throw new CatalogException("Failed to list BigQuery databases (datasets)", e);
+ }
+ return databases;
+ }
+
+ @Override
+ public List<String> listTables(String databaseName)
+ throws CatalogException, DatabaseNotExistException {
+ if (databaseName == null || databaseName.trim().isEmpty()) {
+ databaseName = config.get(BigQuerySinkOptions.DATASET_ID);
+ }
+ if (!databaseExists(databaseName)) {
+ throw new DatabaseNotExistException(catalogName, databaseName);
+ }
+ List<String> tables = new ArrayList<>();
+ try {
+ Page<Table> bqTables = bigquery.listTables(databaseName);
+ for (Table table : bqTables.iterateAll()) {
+ tables.add(table.getTableId().getTable());
+ }
+ } catch (Exception e) {
+ throw new CatalogException("Failed to list tables in dataset: " + databaseName, e);
+ }
+ return tables;
+ }
+
+ @Override
+ public boolean tableExists(TablePath tablePath) throws CatalogException {
+ TableId tableId = TableId.of(getDatasetName(tablePath), tablePath.getTableName());
+ return bigquery.getTable(tableId) != null;
+ }
+
+ @Override
+ public CatalogTable getTable(TablePath tablePath)
+ throws CatalogException, TableNotExistException {
+ TableId tableId = TableId.of(getDatasetName(tablePath), tablePath.getTableName());
+ Table table = bigquery.getTable(tableId);
+ if (table == null) {
+ throw new TableNotExistException(catalogName, tablePath);
+ }
+
+ Schema bqSchema = table.getDefinition().getSchema();
+ TableSchema.Builder schemaBuilder = TableSchema.builder();
+
+ if (bqSchema != null && bqSchema.getFields() != null) {
+ for (Field field : bqSchema.getFields()) {
+ // Skip change capture metadata columns to keep source/target comparison clean
+ if (CHANGE_TYPE.equals(field.getName()) || SEQUENCE_NUM.equals(field.getName())) {
+ continue;
+ }
+ SeaTunnelDataType<?> type = mapToSeaTunnelType(field);
+ Column column =
+ PhysicalColumn.of(
+ field.getName(),
+ type,
+ 0,
+ field.getMode() == Field.Mode.NULLABLE || field.getMode() == null,
+ null,
+ null);
+ schemaBuilder.column(column);
+ }
+ }
+
+ return CatalogTable.of(
+ TableIdentifier.of(catalogName, tablePath),
+ schemaBuilder.build(),
+ Collections.emptyMap(),
+ Collections.emptyList(),
+ "BigQuery Catalog Table");
+ }
+
+ @Override
+ public void createTable(TablePath tablePath, CatalogTable table, boolean ignoreIfExists)
+ throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
+ if (!databaseExists(getDatasetName(tablePath))) {
+ throw new DatabaseNotExistException(catalogName, getDatasetName(tablePath));
+ }
+ if (tableExists(tablePath)) {
+ if (ignoreIfExists) {
+ return;
+ }
+ throw new TableAlreadyExistException(catalogName, tablePath);
+ }
+
+ List<Field> fields = new ArrayList<>();
+ for (Column column : table.getTableSchema().getColumns()) {
+ fields.add(convertColumn(column));
+ }
+
+ // Add stream change capture tracking fields if running in streaming mode
+ boolean isBatch =
+ BigQuerySinkBatchWriter.BATCH.equals(config.get(BigQuerySinkOptions.WRITE_MODE));
+ if (!isBatch) {
+ org.apache.seatunnel.api.table.catalog.PrimaryKey seaTunnelPrimaryKey =
+ table.getTableSchema().getPrimaryKey();
+ if (seaTunnelPrimaryKey == null || seaTunnelPrimaryKey.getColumnNames().isEmpty()) {
+ throw new CatalogException(
+ "Streaming mode requires a Primary Key in the schema of table: "
+ + tablePath.getFullName());
+ }
+ }
+
+ Schema bqSchema = Schema.of(fields);
+ TableId tableId = TableId.of(getDatasetName(tablePath), tablePath.getTableName());
+
+ TableDefinition tableDefinition;
+ org.apache.seatunnel.api.table.catalog.PrimaryKey seaTunnelPrimaryKey =
+ table.getTableSchema().getPrimaryKey();
+
+ if (seaTunnelPrimaryKey != null && !seaTunnelPrimaryKey.getColumnNames().isEmpty()) {
+ com.google.cloud.bigquery.PrimaryKey bqPrimaryKey =
+ com.google.cloud.bigquery.PrimaryKey.newBuilder()
+ .setColumns(seaTunnelPrimaryKey.getColumnNames())
+ .build();
+
+ com.google.cloud.bigquery.TableConstraints constraints =
+ com.google.cloud.bigquery.TableConstraints.newBuilder()
+ .setPrimaryKey(bqPrimaryKey)
+ .build();
+
+ tableDefinition =
+ StandardTableDefinition.newBuilder()
+ .setSchema(bqSchema)
+ .setTableConstraints(constraints)
+ .build();
+ } else {
+ tableDefinition = StandardTableDefinition.of(bqSchema);
+ }
+
+ try {
+ bigquery.create(TableInfo.of(tableId, tableDefinition));
+ log.info("BigQuery Table '{}' created successfully.", tablePath.getFullName());
+ } catch (Exception e) {
+ throw new CatalogException(
+ "Failed to create BigQuery table: " + tablePath.getFullName(), e);
+ }
+ }
+
+ @Override
+ public void dropTable(TablePath tablePath, boolean ignoreIfNotExists)
+ throws TableNotExistException, CatalogException {
+ TableId tableId = TableId.of(getDatasetName(tablePath), tablePath.getTableName());
+ try {
+ boolean deleted = bigquery.delete(tableId);
+ if (!deleted && !ignoreIfNotExists) {
+ throw new TableNotExistException(catalogName, tablePath);
+ }
+ log.info("BigQuery Table '{}' dropped successfully.", tablePath.getFullName());
+ } catch (TableNotExistException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new CatalogException(
+ "Failed to drop BigQuery table: " + tablePath.getFullName(), e);
+ }
+ }
+
+ @Override
+ public void createDatabase(TablePath tablePath, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ String databaseName = getDatasetName(tablePath);
+ if (databaseExists(databaseName)) {
+ if (ignoreIfExists) {
+ return;
+ }
+ throw new DatabaseAlreadyExistException(catalogName, databaseName);
+ }
+ try {
+ bigquery.create(DatasetInfo.newBuilder(databaseName).build());
+ log.info("BigQuery Dataset (database) '{}' created successfully.", databaseName);
+ } catch (Exception e) {
+ throw new CatalogException("Failed to create BigQuery dataset: " + databaseName, e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(TablePath tablePath, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ String databaseName = getDatasetName(tablePath);
+ if (!databaseExists(databaseName)) {
+ if (ignoreIfNotExists) {
+ return;
+ }
+ throw new DatabaseNotExistException(catalogName, databaseName);
+ }
+ try {
+ bigquery.delete(databaseName, BigQuery.DatasetDeleteOption.deleteContents());
+ log.info("BigQuery Dataset (database) '{}' dropped successfully.", databaseName);
+ } catch (Exception e) {
+ throw new CatalogException("Failed to drop BigQuery dataset: " + databaseName, e);
+ }
+ }
+
+ @Override
+ public void truncateTable(TablePath tablePath, boolean ignoreIfNotExists)
+ throws TableNotExistException, CatalogException {
+ if (!tableExists(tablePath)) {
+ if (ignoreIfNotExists) {
+ return;
+ }
+ throw new TableNotExistException(catalogName, tablePath);
+ }
+ String query =
+ String.format(
+ "TRUNCATE TABLE `%s.%s` ;",
+ getDatasetName(tablePath), tablePath.getTableName());
+ QueryJobConfiguration queryConfig = QueryJobConfiguration.newBuilder(query).build();
+ try {
+ bigquery.query(queryConfig);
+ log.info("BigQuery Table '{}' truncated successfully.", tablePath.getFullName());
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new CatalogException("Truncate table interrupted", e);
+ } catch (Exception e) {
+ throw new CatalogException(
+ "Failed to truncate BigQuery table: " + tablePath.getFullName(), e);
+ }
+ }
+
+ @Override
+ public boolean isExistsData(TablePath tablePath) {
+ if (!tableExists(tablePath)) {
+ return false;
+ }
+ String query =
+ String.format(
+ "SELECT 1 FROM `%s.%s` LIMIT 1 ;",
+ getDatasetName(tablePath), tablePath.getTableName());
+ QueryJobConfiguration queryConfig = QueryJobConfiguration.newBuilder(query).build();
+ try {
+ return bigquery.query(queryConfig).iterateAll().iterator().hasNext();
+ } catch (Exception e) {
+ log.warn(
+ "Failed to check if table has data via query, falling back to metadata: {}",
+ e.getMessage());
+ TableId tableId = TableId.of(getDatasetName(tablePath), tablePath.getTableName());
+ Table table = bigquery.getTable(tableId);
+ return table != null && table.getNumRows().longValue() > 0;
+ }
+ }
+
+ @Override
+ public void executeSql(TablePath tablePath, String sql) {
+ if (sql == null || sql.trim().isEmpty()) {
+ log.warn("No custom SQL query provided for table {}, skipping execution.", tablePath);
+ return;
+ }
+ log.info("Executing custom SQL in dataset {}: {}", getDatasetName(tablePath), sql);
+ QueryJobConfiguration queryConfig = QueryJobConfiguration.newBuilder(sql).build();
+ try {
+ bigquery.query(queryConfig);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new CatalogException("Custom SQL execution interrupted", e);
+ } catch (Exception e) {
+ throw new CatalogException("Failed to execute custom SQL", e);
+ }
+ }
+
+ private Field convertColumn(Column column) {
+ StandardSQLTypeName bqType = mapToBigQueryType(column.getDataType().getSqlType());
+ Field.Builder fieldBuilder = Field.newBuilder(column.getName(), bqType);
+
+ if (column.isNullable()) {
+ fieldBuilder.setMode(Field.Mode.NULLABLE);
+ } else {
+ fieldBuilder.setMode(Field.Mode.REQUIRED);
+ }
+
+ if (column.getDataType().getSqlType() == SqlType.ROW) {
+ SeaTunnelRowType rowType = (SeaTunnelRowType) column.getDataType();
+ List<Field> subFields = new ArrayList<>();
+ for (int i = 0; i < rowType.getFieldNames().length; i++) {
+ Column subColumn =
+ PhysicalColumn.of(
+ rowType.getFieldNames()[i],
+ rowType.getFieldType(i),
+ 0,
+ true,
+ null,
+ null);
+ subFields.add(convertColumn(subColumn));
+ }
+ fieldBuilder.setType(bqType, FieldList.of(subFields));
+ }
+
+ return fieldBuilder.build();
+ }
+
+ private StandardSQLTypeName mapToBigQueryType(SqlType sqlType) {
+ switch (sqlType) {
+ case BOOLEAN:
+ return StandardSQLTypeName.BOOL;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ return StandardSQLTypeName.INT64;
+ case FLOAT:
+ case DOUBLE:
+ return StandardSQLTypeName.FLOAT64;
+ case DECIMAL:
+ return StandardSQLTypeName.NUMERIC;
+ case DATE:
+ return StandardSQLTypeName.DATE;
+ case TIME:
+ return StandardSQLTypeName.TIME;
+ case TIMESTAMP:
+ return StandardSQLTypeName.TIMESTAMP;
+ case BYTES:
+ return StandardSQLTypeName.BYTES;
+ case ROW:
+ return StandardSQLTypeName.STRUCT;
+ case STRING:
+ default:
+ return StandardSQLTypeName.STRING;
+ }
+ }
+
+ private SeaTunnelDataType<?> mapToSeaTunnelType(Field field) {
+ StandardSQLTypeName standardType = field.getType().getStandardType();
+ switch (standardType) {
+ case BOOL:
+ return BasicType.BOOLEAN_TYPE;
+ case INT64:
+ return BasicType.LONG_TYPE;
+ case FLOAT64:
+ return BasicType.DOUBLE_TYPE;
+ case NUMERIC:
+ case BIGNUMERIC:
+ return new DecimalType(38, 9);
+ case BYTES:
+ return PrimitiveByteArrayType.INSTANCE;
+ case DATE:
+ return LocalTimeType.LOCAL_DATE_TYPE;
+ case TIME:
+ return LocalTimeType.LOCAL_TIME_TYPE;
+ case DATETIME:
+ case TIMESTAMP:
+ return LocalTimeType.LOCAL_DATE_TIME_TYPE;
+ case STRUCT:
+ FieldList subFields = field.getSubFields();
+ String[] fieldNames = new String[subFields.size()];
+ SeaTunnelDataType<?>[] fieldTypes = new SeaTunnelDataType<?>[subFields.size()];
+ for (int i = 0; i < subFields.size(); i++) {
+ Field subField = subFields.get(i);
+ fieldNames[i] = subField.getName();
+ fieldTypes[i] = mapToSeaTunnelType(subField);
+ }
+ return new SeaTunnelRowType(fieldNames, fieldTypes);
+ case STRING:
+ default:
+ return BasicType.STRING_TYPE;
+ }
+ }
+}
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/client/BigQueryClientFactory.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/client/BigQueryClientFactory.java
index 5f31756fe6..a6d770def7 100644
--- a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/client/BigQueryClientFactory.java
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/client/BigQueryClientFactory.java
@@ -71,11 +71,15 @@ public class BigQueryClientFactory {
GoogleCredentials credentials = getCredentials(config);
- BigQueryWriteSettings settings =
+ BigQueryWriteSettings.Builder settingsBuilder =
BigQueryWriteSettings.newBuilder()
- .setCredentialsProvider(FixedCredentialsProvider.create(credentials))
- .build();
- return BigQueryWriteClient.create(settings);
+ .setCredentialsProvider(FixedCredentialsProvider.create(credentials));
+
+ if (config.get(BigQuerySinkOptions.UNIVERSE_DOMAIN) != null) {
+ settingsBuilder.setUniverseDomain(config.get(BigQuerySinkOptions.UNIVERSE_DOMAIN));
+ }
+
+ return BigQueryWriteClient.create(settingsBuilder.build());
} catch (IOException e) {
throw new BigQueryConnectorException(
BigQueryConnectorErrorCode.CLIENT_CREATE_FAILED,
@@ -97,11 +101,14 @@ public class BigQueryClientFactory {
GoogleCredentials credentials = getCredentials(config);
- return BigQueryOptions.newBuilder()
- .setProjectId(projectId)
- .setCredentials(credentials)
- .build()
- .getService();
+ BigQueryOptions.Builder builder =
+ BigQueryOptions.newBuilder().setProjectId(projectId).setCredentials(credentials);
+
+ if (config.get(BigQuerySinkOptions.UNIVERSE_DOMAIN) != null) {
+ builder.setUniverseDomain(config.get(BigQuerySinkOptions.UNIVERSE_DOMAIN));
+ }
+
+ return builder.build().getService();
}
public static GoogleCredentials getCredentials(ReadonlyConfig config) {
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/option/BigQuerySinkOptions.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/option/BigQuerySinkOptions.java
index 32b69960c4..8dc3e7b6ab 100644
--- a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/option/BigQuerySinkOptions.java
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/option/BigQuerySinkOptions.java
@@ -19,6 +19,8 @@ package org.apache.seatunnel.connectors.bigquery.option;
import org.apache.seatunnel.api.configuration.Option;
import org.apache.seatunnel.api.configuration.Options;
+import org.apache.seatunnel.api.sink.DataSaveMode;
+import org.apache.seatunnel.api.sink.SchemaSaveMode;
public class BigQuerySinkOptions {
@@ -80,4 +82,29 @@ public class BigQuerySinkOptions {
.noDefaultValue()
.withDescription(
"The host of the BigQuery emulator (e.g. localhost:9050). Only for testing purposes.");
+
+ public static final Option<String> UNIVERSE_DOMAIN =
+ Options.key("universe_domain")
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ "The Google Cloud Universe Domain (e.g. s3nsapis.fr for S3NS sovereign cloud)");
+
+ public static final Option<SchemaSaveMode> SCHEMA_SAVE_MODE =
+ Options.key("schema_save_mode")
+ .enumType(SchemaSaveMode.class)
+ .defaultValue(SchemaSaveMode.CREATE_SCHEMA_WHEN_NOT_EXIST)
+ .withDescription("schema save mode");
+
+ public static final Option<DataSaveMode> DATA_SAVE_MODE =
+ Options.key("data_save_mode")
+ .enumType(DataSaveMode.class)
+ .defaultValue(DataSaveMode.APPEND_DATA)
+ .withDescription("data save mode");
+
+ public static final Option<String> CUSTOM_SQL =
+ Options.key("custom_sql")
+ .stringType()
+ .noDefaultValue()
+ .withDescription("custom SQL to execute for CUSTOM_PROCESSING data save mode");
}
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/AbstractBigQuerySinkWriter.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/AbstractBigQuerySinkWriter.java
index e49c8dcf40..ad0fdac8d5 100644
--- a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/AbstractBigQuerySinkWriter.java
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/AbstractBigQuerySinkWriter.java
@@ -18,9 +18,11 @@
package org.apache.seatunnel.connectors.bigquery.sink;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.sink.MultiTableResourceManager;
import org.apache.seatunnel.api.sink.SinkWriter;
import org.apache.seatunnel.api.sink.SupportMultiTableSinkWriter;
import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.connectors.bigquery.client.BigQueryClientFactory;
import org.apache.seatunnel.connectors.bigquery.convert.BigQuerySerializer;
import org.apache.seatunnel.connectors.bigquery.exception.BigQueryConnectorErrorCode;
import org.apache.seatunnel.connectors.bigquery.exception.BigQueryConnectorException;
@@ -40,29 +42,40 @@ import java.util.concurrent.TimeUnit;
@Slf4j
public abstract class AbstractBigQuerySinkWriter
implements SinkWriter<SeaTunnelRow, BigQueryCommitInfo, BigQuerySinkState>,
- SupportMultiTableSinkWriter<Void> {
+ SupportMultiTableSinkWriter<BigQueryWriteClient> {
protected final ReadonlyConfig config;
protected final BigQuerySerializer serializer;
- protected final BigQueryWriteClient client;
+ protected BigQueryWriteClient client;
protected BigQueryWriter streamWriter;
protected final int batchSize;
protected JSONArray buffer = new JSONArray();
+ protected AbstractBigQuerySinkWriter(
+ ReadonlyConfig readOnlyConfig, BigQuerySerializer serializer) {
+ this.config = readOnlyConfig;
+ this.batchSize = readOnlyConfig.get(BigQuerySinkOptions.BATCH_SIZE);
+ this.serializer = serializer;
+ }
+
protected AbstractBigQuerySinkWriter(
ReadonlyConfig readOnlyConfig,
BigQueryWriter streamWriter,
- BigQuerySerializer serializer,
- BigQueryWriteClient client) {
+ BigQuerySerializer serializer) {
this.config = readOnlyConfig;
this.batchSize = readOnlyConfig.get(BigQuerySinkOptions.BATCH_SIZE);
this.streamWriter = streamWriter;
this.serializer = serializer;
- this.client = client;
+ }
+
+ @Override
+ public MultiTableResourceManager<BigQueryWriteClient> initMultiTableResourceManager(
+ int tableSize, int queueSize) {
+ return new BigQueryMultiTableResourceManager(BigQueryClientFactory.getWriteClient(config));
}
protected void flush() {
- if (buffer.length() == 0) return;
+ if (streamWriter == null || buffer.length() == 0) return;
JSONArray dataToSend = buffer;
buffer = new JSONArray();
@@ -94,15 +107,12 @@ public abstract class AbstractBigQuerySinkWriter
}
} finally {
try {
- streamWriter.close();
+ if (streamWriter != null) {
+ streamWriter.close();
+ }
} catch (Exception e) {
log.warn("Failed to close streamWriter", e);
}
- try {
- client.close();
- } catch (Exception e) {
- log.warn("Failed to close BigQueryWriteClient", e);
- }
}
}
}
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQueryMultiTableResourceManager.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQueryMultiTableResourceManager.java
new file mode 100644
index 0000000000..018bf805f0
--- /dev/null
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQueryMultiTableResourceManager.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.bigquery.sink;
+
+import org.apache.seatunnel.api.sink.MultiTableResourceManager;
+
+import com.google.cloud.bigquery.storage.v1.BigQueryWriteClient;
+import lombok.extern.slf4j.Slf4j;
+
+import java.util.Optional;
+
+/**
+ * Resource manager for coordinating and sharing a single {@link BigQueryWriteClient} across
+ * multiple sink writers to optimize connection usage and simplify resource cleanup.
+ */
+@Slf4j
+public class BigQueryMultiTableResourceManager
+ implements MultiTableResourceManager<BigQueryWriteClient> {
+
+ private final BigQueryWriteClient client;
+
+ /**
+ * Constructs a BigQueryMultiTableResourceManager with the specified shared client.
+ *
+ * @param client the shared {@link BigQueryWriteClient} instance
+ */
+ public BigQueryMultiTableResourceManager(BigQueryWriteClient client) {
+ this.client = client;
+ }
+
+ /**
+ * Retrieves the shared {@link BigQueryWriteClient} resource.
+ *
+ * @return an {@link Optional} containing the shared client if present
+ */
+ @Override
+ public Optional<BigQueryWriteClient> getSharedResource() {
+ return Optional.ofNullable(client);
+ }
+
+ @Override
+ public void close() {
+ if (client != null) {
+ log.info("Closing shared worker BigQueryWriteClient connection...");
+ try {
+ client.close();
+ } catch (Exception e) {
+ log.warn("Failed to close shared BigQueryWriteClient", e);
+ }
+ }
+ }
+}
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySaveModeHandler.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySaveModeHandler.java
new file mode 100644
index 0000000000..e29459fb9a
--- /dev/null
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySaveModeHandler.java
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.bigquery.sink;
+
+import org.apache.seatunnel.api.sink.DataSaveMode;
+import org.apache.seatunnel.api.sink.DefaultSaveModeHandler;
+import org.apache.seatunnel.api.sink.SchemaSaveMode;
+import org.apache.seatunnel.api.table.catalog.Catalog;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.exception.CatalogException;
+import org.apache.seatunnel.api.table.catalog.exception.TableNotExistException;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+
+import com.google.cloud.bigquery.BigQueryException;
+import lombok.extern.slf4j.Slf4j;
+
+import java.util.Optional;
+
+/**
+ * Handler for managing BigQuery schema and data save modes. This class validates schema coherence
+ * between local schemas and remote BigQuery schemas, and handles database/table life cycle based on
+ * configurations.
+ */
+@Slf4j
+public class BigQuerySaveModeHandler extends DefaultSaveModeHandler {
+
+ /**
+ * Constructs a new BigQuerySaveModeHandler.
+ *
+ * @param schemaSaveMode the schema save mode configuration
+ * @param dataSaveMode the data save mode configuration
+ * @param catalog the BigQuery catalog instance
+ * @param tablePath the path of the target table
+ * @param catalogTable the local catalog table schema representation
+ * @param customSql optional custom SQL script to run during save mode handling
+ */
+ public BigQuerySaveModeHandler(
+ SchemaSaveMode schemaSaveMode,
+ DataSaveMode dataSaveMode,
+ Catalog catalog,
+ TablePath tablePath,
+ CatalogTable catalogTable,
+ String customSql) {
+ super(schemaSaveMode, dataSaveMode, catalog, tablePath, catalogTable, customSql);
+ }
+
+ /**
+ * Executes the schema save mode operations (e.g., table deletion, creation, recreation) and
+ * performs schema coherence check to ensure the target table is compatible.
+ */
+ @Override
+ public void handleSchemaSaveMode() {
+ // 1. Let the default handler execute table creation / recreation / deletion
+ super.handleSchemaSaveMode();
+
+ // 2. Perform schema coherence validation
+ validateSchemaCoherence();
+ }
+
+ /**
+ * Validates that the schema of the local table matches or is compatible with the remote
+ * BigQuery table's schema.
+ *
+ * @throws CatalogException if schema coherence validation fails or Google SDK exception is
+ * caught
+ * @throws SeaTunnelRuntimeException if any system runtime exception is encountered
+ */
+ private void validateSchemaCoherence() {
+ if (schemaSaveMode == SchemaSaveMode.RECREATE_SCHEMA
+ || schemaSaveMode == SchemaSaveMode.IGNORE) {
+ return;
+ }
+
+ try {
+ CatalogTable remoteTable = catalog.getTable(tablePath);
+ if (remoteTable == null || remoteTable.getTableSchema() == null) {
+ return;
+ }
+
+ for (Column sourceColumn : catalogTable.getTableSchema().getColumns()) {
+ Optional<Column> remoteColOpt =
+ remoteTable.getTableSchema().getColumns().stream()
+ .filter(c -> c.getName().equalsIgnoreCase(sourceColumn.getName()))
+ .findFirst();
+
+ if (!remoteColOpt.isPresent()) {
+ throw new CatalogException(
+ String.format(
+ "Target BigQuery table '%s' is missing source column: '%s'",
+ tablePath.getFullName(), sourceColumn.getName()));
+ }
+
+ Column remoteColumn = remoteColOpt.get();
+ if (!isTypeCompatible(
+ sourceColumn.getDataType().getSqlType(),
+ remoteColumn.getDataType().getSqlType())) {
+ throw new CatalogException(
+ String.format(
+ "Type mismatch for column '%s' inside target BigQuery table '%s'. Source type: '%s', target type: '%s'",
+ sourceColumn.getName(),
+ tablePath.getFullName(),
+ sourceColumn.getDataType().getSqlType(),
+ remoteColumn.getDataType().getSqlType()));
+ }
+ }
+ log.info(
+ "BigQuery schema coherence check passed successfully for table: {}",
+ tablePath.getFullName());
+ } catch (TableNotExistException e) {
+ log.info(
+ "Target BigQuery table '{}' does not exist yet. Skipping schema coherence check.",
+ tablePath.getFullName());
+ return;
+ } catch (Exception e) {
+ if (e instanceof CatalogException) {
+ throw (CatalogException) e;
+ }
+ if (e instanceof SeaTunnelRuntimeException) {
+ throw (SeaTunnelRuntimeException) e;
+ }
+ if (e instanceof BigQueryException) {
+ throw new CatalogException("Failed to validate BigQuery schema coherence", e);
+ }
+ log.warn("Schema validation check ignored due to exception: {}", e.getMessage());
+ }
+ }
+
+ private boolean isTypeCompatible(SqlType source, SqlType sink) {
+ if (source == sink) {
+ return true;
+ }
+ // Widening integer compatibility
+ if ((source == SqlType.TINYINT
+ || source == SqlType.SMALLINT
+ || source == SqlType.INT
+ || source == SqlType.BIGINT)
+ && (sink == SqlType.BIGINT)) {
+ return true;
+ }
+ // Widening float compatibility
+ if ((source == SqlType.FLOAT || source == SqlType.DOUBLE) && (sink == SqlType.DOUBLE)) {
+ return true;
+ }
+ return false;
+ }
+}
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySink.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySink.java
index 4b0457a9fd..fb9e5b1d9a 100644
--- a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySink.java
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySink.java
@@ -19,23 +19,25 @@ package org.apache.seatunnel.connectors.bigquery.sink;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.api.serialization.Serializer;
+import org.apache.seatunnel.api.sink.DataSaveMode;
+import org.apache.seatunnel.api.sink.SaveModeHandler;
+import org.apache.seatunnel.api.sink.SchemaSaveMode;
import org.apache.seatunnel.api.sink.SeaTunnelSink;
import org.apache.seatunnel.api.sink.SinkCommitter;
import org.apache.seatunnel.api.sink.SinkWriter;
+import org.apache.seatunnel.api.sink.SupportMultiTableSink;
+import org.apache.seatunnel.api.sink.SupportSaveMode;
+import org.apache.seatunnel.api.table.catalog.Catalog;
import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.TablePath;
import org.apache.seatunnel.api.table.type.SeaTunnelRow;
import org.apache.seatunnel.common.exception.CommonError;
-import org.apache.seatunnel.connectors.bigquery.client.BigQueryClientFactory;
+import org.apache.seatunnel.connectors.bigquery.catalog.BigQueryCatalog;
import org.apache.seatunnel.connectors.bigquery.convert.BigQuerySerializer;
import org.apache.seatunnel.connectors.bigquery.option.BigQuerySinkOptions;
import org.apache.seatunnel.connectors.bigquery.sink.committer.BigQueryCommitInfo;
import org.apache.seatunnel.connectors.bigquery.sink.committer.BigQueryCommitInfoSerializer;
import org.apache.seatunnel.connectors.bigquery.sink.committer.BigQueryCommitter;
-import org.apache.seatunnel.connectors.bigquery.sink.writer.BigQueryBatchWriter;
-import org.apache.seatunnel.connectors.bigquery.sink.writer.BigQueryStreamWriter;
-import org.apache.seatunnel.connectors.bigquery.sink.writer.BigQueryWriter;
-
-import com.google.cloud.bigquery.storage.v1.BigQueryWriteClient;
import java.util.Comparator;
import java.util.List;
@@ -43,7 +45,9 @@ import java.util.Optional;
public class BigQuerySink
implements SeaTunnelSink<
- SeaTunnelRow, BigQuerySinkState, BigQueryCommitInfo, BigQueryCommitInfo> {
+ SeaTunnelRow, BigQuerySinkState, BigQueryCommitInfo, BigQueryCommitInfo>,
+ SupportSaveMode,
+ SupportMultiTableSink {
private final ReadonlyConfig config;
private final boolean isBatch;
@@ -66,48 +70,33 @@ public class BigQuerySink
@Override
public AbstractBigQuerySinkWriter createWriter(SinkWriter.Context context) {
- BigQueryWriteClient client = BigQueryClientFactory.getWriteClient(config);
if (isBatch) {
return new BigQuerySinkBatchWriter(
- config,
- BigQueryBatchWriter.of(client, config),
- new BigQuerySerializer(catalogTable, config),
- client);
+ config, new BigQuerySerializer(catalogTable, config));
} else {
return new BigQuerySinkStreamWriter(
- config,
- BigQueryStreamWriter.of(client, config),
- new BigQuerySerializer(catalogTable, config),
- client);
+ config, new BigQuerySerializer(catalogTable, config));
}
}
@Override
public SinkWriter<SeaTunnelRow, BigQueryCommitInfo, BigQuerySinkState> restoreWriter(
SinkWriter.Context context, List<BigQuerySinkState> states) {
- BigQueryWriteClient client = BigQueryClientFactory.getWriteClient(config);
- BigQueryWriter writer;
if (isBatch) {
if (states != null && !states.isEmpty()) {
BigQuerySinkState latestState = getLatestState(states);
- writer =
- BigQueryBatchWriter.restore(
- client,
- config,
- latestState.getStreamName(),
- latestState.getNextOffset());
+ return new BigQuerySinkBatchWriter(
+ config,
+ new BigQuerySerializer(catalogTable, config),
+ latestState.getStreamName(),
+ latestState.getNextOffset());
} else {
- writer = BigQueryBatchWriter.of(client, config);
+ return new BigQuerySinkBatchWriter(
+ config, new BigQuerySerializer(catalogTable, config));
}
-
- return new BigQuerySinkBatchWriter(
- config, writer, new BigQuerySerializer(catalogTable, config), client);
} else {
return new BigQuerySinkStreamWriter(
- config,
- BigQueryStreamWriter.of(client, config),
- new BigQuerySerializer(catalogTable, config),
- client);
+ config, new BigQuerySerializer(catalogTable, config));
}
}
@@ -136,6 +125,27 @@ public class BigQuerySink
return BigQuerySinkOptions.IDENTIFIER;
}
+ @Override
+ public Optional<SaveModeHandler> getSaveModeHandler() {
+ if (catalogTable == null) {
+ return Optional.empty();
+ }
+ Catalog catalog = new BigQueryCatalog(catalogTable.getCatalogName(), config);
+ SchemaSaveMode schemaSaveMode = config.get(BigQuerySinkOptions.SCHEMA_SAVE_MODE);
+ DataSaveMode dataSaveMode = config.get(BigQuerySinkOptions.DATA_SAVE_MODE);
+ TablePath tablePath = catalogTable.getTableId().toTablePath();
+ String configuredTableId = config.get(BigQuerySinkOptions.TABLE_ID);
+ // Note: config option TABLE_ID is already placeholder-resolved per target table
+ // by TablePlaceholderProcessor during FactoryUtil.createAndPrepareSink()
+ if (configuredTableId != null && !configuredTableId.contains("${table_name}")) {
+ tablePath = TablePath.of(tablePath.getDatabaseName(), configuredTableId);
+ }
+ String customSql = config.get(BigQuerySinkOptions.CUSTOM_SQL);
+ return Optional.of(
+ new BigQuerySaveModeHandler(
+ schemaSaveMode, dataSaveMode, catalog, tablePath, catalogTable, customSql));
+ }
+
static BigQuerySinkState getLatestState(List<BigQuerySinkState> states) {
return states.stream()
.max(Comparator.comparingLong(BigQuerySinkState::getCheckpointId))
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkBatchWriter.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkBatchWriter.java
index f9d13449d4..674db17490 100644
--- a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkBatchWriter.java
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkBatchWriter.java
@@ -18,6 +18,7 @@
package org.apache.seatunnel.connectors.bigquery.sink;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.sink.MultiTableResourceManager;
import org.apache.seatunnel.api.table.type.SeaTunnelRow;
import org.apache.seatunnel.connectors.bigquery.convert.BigQuerySerializer;
import org.apache.seatunnel.connectors.bigquery.exception.BigQueryConnectorErrorCode;
@@ -45,12 +46,44 @@ import java.util.concurrent.TimeUnit;
public class BigQuerySinkBatchWriter extends AbstractBigQuerySinkWriter {
public static final String BATCH = "batch";
+ private final String restoredStreamName;
+ private final long restoredNextOffset;
+
+ public BigQuerySinkBatchWriter(ReadonlyConfig readOnlyConfig, BigQuerySerializer serializer) {
+ this(readOnlyConfig, serializer, null, 0L);
+ }
+
public BigQuerySinkBatchWriter(
ReadonlyConfig readOnlyConfig,
- BigQueryWriter streamWriter,
BigQuerySerializer serializer,
- BigQueryWriteClient client) {
- super(readOnlyConfig, streamWriter, serializer, client);
+ String restoredStreamName,
+ long restoredNextOffset) {
+ super(readOnlyConfig, serializer);
+ this.restoredStreamName = restoredStreamName;
+ this.restoredNextOffset = restoredNextOffset;
+ }
+
+ public BigQuerySinkBatchWriter(
+ ReadonlyConfig readOnlyConfig,
+ BigQueryWriter streamWriter,
+ BigQuerySerializer serializer) {
+ super(readOnlyConfig, streamWriter, serializer);
+ this.restoredStreamName = null;
+ this.restoredNextOffset = 0L;
+ }
+
+ @Override
+ public void setMultiTableResourceManager(
+ MultiTableResourceManager<BigQueryWriteClient> manager, int queueIndex) {
+ log.info("Injecting shared client and initializing Batch stream writer...");
+ this.client = manager.getSharedResource().get();
+ if (restoredStreamName != null) {
+ this.streamWriter =
+ BigQueryBatchWriter.restore(
+ client, config, restoredStreamName, restoredNextOffset);
+ } else {
+ this.streamWriter = BigQueryBatchWriter.of(client, config);
+ }
}
@Override
@@ -164,7 +197,7 @@ public class BigQuerySinkBatchWriter extends AbstractBigQuerySinkWriter {
// Batch mode uses BigQuery buffered streams and stores streamName + nextOffset
// in checkpoint state. Flushing during close could append rows outside the
// latest checkpoint state and make the external stream offset move ahead of
- // the restored nextOffset.
+ // the restored nextOffset, breaking 2PC state recovery contract.
return false;
}
}
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkFactory.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkFactory.java
index 82970d648c..2e11e4bf41 100644
--- a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkFactory.java
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkFactory.java
@@ -52,6 +52,7 @@ public class BigQuerySinkFactory implements TableSinkFactory {
BigQuerySinkOptions.SEQUENCE_NUMBER_COLUMN,
BigQuerySinkOptions.BATCH_SIZE,
BigQuerySinkOptions.EMULATOR_HOST,
+ BigQuerySinkOptions.UNIVERSE_DOMAIN,
SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA)
.build();
}
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkStreamWriter.java b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkStreamWriter.java
index cd7c70dceb..696923c5d1 100644
--- a/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkStreamWriter.java
+++ b/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkStreamWriter.java
@@ -18,9 +18,11 @@
package org.apache.seatunnel.connectors.bigquery.sink;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.sink.MultiTableResourceManager;
import org.apache.seatunnel.api.table.type.SeaTunnelRow;
import org.apache.seatunnel.connectors.bigquery.convert.BigQuerySerializer;
import org.apache.seatunnel.connectors.bigquery.sink.committer.BigQueryCommitInfo;
+import org.apache.seatunnel.connectors.bigquery.sink.writer.BigQueryStreamWriter;
import org.apache.seatunnel.connectors.bigquery.sink.writer.BigQueryWriter;
import com.google.cloud.bigquery.storage.v1.BigQueryWriteClient;
@@ -32,12 +34,23 @@ import java.util.Optional;
public class BigQuerySinkStreamWriter extends AbstractBigQuerySinkWriter {
public static final String STREAMING = "streaming";
+ public BigQuerySinkStreamWriter(ReadonlyConfig readOnlyConfig, BigQuerySerializer serializer) {
+ super(readOnlyConfig, serializer);
+ }
+
public BigQuerySinkStreamWriter(
ReadonlyConfig readOnlyConfig,
BigQueryWriter streamWriter,
- BigQuerySerializer serializer,
- BigQueryWriteClient client) {
- super(readOnlyConfig, streamWriter, serializer, client);
+ BigQuerySerializer serializer) {
+ super(readOnlyConfig, streamWriter, serializer);
+ }
+
+ @Override
+ public void setMultiTableResourceManager(
+ MultiTableResourceManager<BigQueryWriteClient> manager, int queueIndex) {
+ log.info("Injecting shared client and initializing Streaming stream writer...");
+ this.client = manager.getSharedResource().get();
+ this.streamWriter = BigQueryStreamWriter.of(client, config);
}
@Override
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQueryCatalogAndSaveModeTest.java b/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQueryCatalogAndSaveModeTest.java
new file mode 100644
index 0000000000..95daae13d7
--- /dev/null
+++ b/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQueryCatalogAndSaveModeTest.java
@@ -0,0 +1,442 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.bigquery.sink;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.sink.DataSaveMode;
+import org.apache.seatunnel.api.sink.SchemaSaveMode;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.catalog.exception.CatalogException;
+import org.apache.seatunnel.api.table.catalog.exception.TableAlreadyExistException;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.connectors.bigquery.catalog.BigQueryCatalog;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.Dataset;
+import com.google.cloud.bigquery.Field;
+import com.google.cloud.bigquery.StandardSQLTypeName;
+import com.google.cloud.bigquery.Table;
+import com.google.cloud.bigquery.TableDefinition;
+import com.google.cloud.bigquery.TableId;
+import com.google.cloud.bigquery.TableInfo;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class BigQueryCatalogAndSaveModeTest {
+
+ @Mock private BigQuery mockBigQuery;
+ private BigQueryCatalog catalog;
+ private ReadonlyConfig mockConfig;
+
+ @BeforeEach
+ public void setup() throws Exception {
+ MockitoAnnotations.openMocks(this);
+ Map<String, Object> configMap = new HashMap<>();
+ configMap.put("project_id", "test-project");
+ configMap.put("dataset_id", "test-dataset");
+ configMap.put("write_mode", "batch");
+ mockConfig = ReadonlyConfig.fromMap(configMap);
+
+ catalog = new BigQueryCatalog("test-catalog", mockConfig);
+
+ // Inject the mocked BigQuery client into the catalog using reflection
+ java.lang.reflect.Field clientField = BigQueryCatalog.class.getDeclaredField("bigquery");
+ clientField.setAccessible(true);
+ clientField.set(catalog, mockBigQuery);
+ }
+
+ @Test
+ public void testDatabaseExists() {
+ when(mockBigQuery.getDataset("existing-db")).thenReturn(mock(Dataset.class));
+ when(mockBigQuery.getDataset("non-existing-db")).thenReturn(null);
+
+ assertTrue(catalog.databaseExists("existing-db"));
+ assertTrue(!catalog.databaseExists("non-existing-db"));
+ }
+
+ @Test
+ public void testTableExists() {
+ TablePath tablePath = TablePath.of("test-dataset", "test-table");
+ TableId tableId = TableId.of("test-dataset", "test-table");
+ when(mockBigQuery.getTable(tableId)).thenReturn(mock(Table.class));
+
+ assertTrue(catalog.tableExists(tablePath));
+ }
+
+ @Test
+ public void testCreateTableCase1_NewTable() throws Exception {
+ TablePath tablePath = TablePath.of("test-dataset", "new-table");
+ TableId tableId = TableId.of("test-dataset", "new-table");
+ CatalogTable catalogTable = createMockCatalogTable(tablePath);
+
+ when(mockBigQuery.getDataset("test-dataset")).thenReturn(mock(Dataset.class));
+ when(mockBigQuery.getTable(tableId)).thenReturn(null);
+
+ catalog.createTable(tablePath, catalogTable, false);
+
+ verify(mockBigQuery, times(1)).create(any(TableInfo.class));
+ }
+
+ @Test
+ public void testCreateTableCase2_ExistingTableThrowsError() {
+ TablePath tablePath = TablePath.of("test-dataset", "existing-table");
+ TableId tableId = TableId.of("test-dataset", "existing-table");
+ CatalogTable catalogTable = createMockCatalogTable(tablePath);
+
+ when(mockBigQuery.getDataset("test-dataset")).thenReturn(mock(Dataset.class));
+ when(mockBigQuery.getTable(tableId)).thenReturn(mock(Table.class));
+
+ assertThrows(
+ TableAlreadyExistException.class,
+ () -> catalog.createTable(tablePath, catalogTable, false));
+ }
+
+ @Test
+ public void testDropTable() throws Exception {
+ TablePath tablePath = TablePath.of("test-dataset", "drop-table");
+ TableId tableId = TableId.of("test-dataset", "drop-table");
+
+ when(mockBigQuery.delete(tableId)).thenReturn(true);
+
+ catalog.dropTable(tablePath, false);
+
+ verify(mockBigQuery, times(1)).delete(tableId);
+ }
+
+ @Test
+ public void testSaveModeHandler_RecreateSchema() {
+ TablePath tablePath = TablePath.of("test-dataset", "recreate-table");
+ TableId tableId = TableId.of("test-dataset", "recreate-table");
+ CatalogTable catalogTable = createMockCatalogTable(tablePath);
+
+ when(mockBigQuery.getDataset("test-dataset")).thenReturn(mock(Dataset.class));
+ when(mockBigQuery.getTable(tableId)).thenReturn(mock(Table.class)).thenReturn(null);
+ when(mockBigQuery.delete(tableId)).thenReturn(true);
+
+ BigQuerySaveModeHandler handler =
+ new BigQuerySaveModeHandler(
+ SchemaSaveMode.RECREATE_SCHEMA,
+ DataSaveMode.APPEND_DATA,
+ catalog,
+ tablePath,
+ catalogTable,
+ null);
+
+ handler.handleSchemaSaveMode();
+
+ // Should drop the existing table and recreate it
+ verify(mockBigQuery, times(1)).delete(tableId);
+ verify(mockBigQuery, times(1)).create(any(TableInfo.class));
+ }
+
+ @Test
+ public void testSaveModeHandler_CreateSchemaWhenNotExist_ExistsSkip() {
+ TablePath tablePath = TablePath.of("test-dataset", "exists-table");
+ TableId tableId = TableId.of("test-dataset", "exists-table");
+ CatalogTable catalogTable = createMockCatalogTable(tablePath);
+
+ when(mockBigQuery.getDataset("test-dataset")).thenReturn(mock(Dataset.class));
+ when(mockBigQuery.getTable(tableId)).thenReturn(mock(Table.class));
+
+ BigQuerySaveModeHandler handler =
+ new BigQuerySaveModeHandler(
+ SchemaSaveMode.CREATE_SCHEMA_WHEN_NOT_EXIST,
+ DataSaveMode.APPEND_DATA,
+ catalog,
+ tablePath,
+ catalogTable,
+ null);
+
+ handler.handleSchemaSaveMode();
+
+ // Should skip both deletion and creation
+ verify(mockBigQuery, never()).delete(tableId);
+ verify(mockBigQuery, never()).create(any(TableInfo.class));
+ }
+
+ @Test
+ public void testSaveModeHandler_CoherenceCheckFailsForTypeMismatch() {
+ TablePath tablePath = TablePath.of("test-dataset", "mismatch-table");
+ TableId tableId = TableId.of("test-dataset", "mismatch-table");
+ CatalogTable catalogTable = createMockCatalogTable(tablePath);
+
+ Table mockTable = mock(Table.class);
+ TableDefinition mockDef = mock(TableDefinition.class);
+ when(mockTable.getDefinition()).thenReturn(mockDef);
+
+ // Remote table has STRING, while source is INT64
+ com.google.cloud.bigquery.Schema remoteSchema =
+ com.google.cloud.bigquery.Schema.of(Field.of("id", StandardSQLTypeName.STRING));
+ when(mockDef.getSchema()).thenReturn(remoteSchema);
+
+ when(mockBigQuery.getDataset("test-dataset")).thenReturn(mock(Dataset.class));
+ when(mockBigQuery.getTable(tableId)).thenReturn(mockTable);
+
+ BigQuerySaveModeHandler handler =
+ new BigQuerySaveModeHandler(
+ SchemaSaveMode.CREATE_SCHEMA_WHEN_NOT_EXIST,
+ DataSaveMode.APPEND_DATA,
+ catalog,
+ tablePath,
+ catalogTable,
+ null);
+
+ // Executing the schema save mode should throw a CatalogException for type incompatibility
+ assertThrows(CatalogException.class, () -> handler.handleSchemaSaveMode());
+ }
+
+ // ────────────────────────────────────────────────────────────────────────
+ // 3D COMBINATORIAL PARAMETERIZED MATRIX TESTING
+ // ────────────────────────────────────────────────────────────────────────
+
+ public enum TableState {
+ NOT_EXIST,
+ EXIST_CORRECT_SCHEMA,
+ EXIST_WRONG_SCHEMA
+ }
+
+ public enum ExpectedOutcome {
+ SUCCESS,
+ THROWS_TABLE_NOT_EXIST,
+ THROWS_COHERENCE_MISMATCH,
+ THROWS_DATA_ALREADY_EXISTS
+ }
+
+ private static Stream<Arguments> saveModeMatrixProvider() {
+ List<Arguments> cases = new ArrayList<>();
+ for (SchemaSaveMode schemaSaveMode : SchemaSaveMode.values()) {
+ for (DataSaveMode dataSaveMode : DataSaveMode.values()) {
+ for (TableState tableState : TableState.values()) {
+ for (boolean isBatch : new boolean[] {true, false}) {
+ ExpectedOutcome outcome =
+ determineExpectedOutcome(schemaSaveMode, dataSaveMode, tableState);
+ cases.add(
+ Arguments.of(
+ schemaSaveMode,
+ dataSaveMode,
+ tableState,
+ isBatch,
+ outcome));
+ }
+ }
+ }
+ }
+ return cases.stream();
+ }
+
+ private static ExpectedOutcome determineExpectedOutcome(
+ SchemaSaveMode schemaSaveMode, DataSaveMode dataSaveMode, TableState tableState) {
+
+ // Phase 1: Determine Schema Stage behavior
+ boolean isNewTableCreated = false;
+
+ if (tableState == TableState.NOT_EXIST) {
+ if (schemaSaveMode == SchemaSaveMode.ERROR_WHEN_SCHEMA_NOT_EXIST) {
+ return ExpectedOutcome.THROWS_TABLE_NOT_EXIST; // Fatal Schema Error
+ }
+ if (schemaSaveMode == SchemaSaveMode.RECREATE_SCHEMA
+ || schemaSaveMode == SchemaSaveMode.CREATE_SCHEMA_WHEN_NOT_EXIST) {
+ isNewTableCreated = true;
+ }
+ } else {
+ // Table exists
+ if (schemaSaveMode == SchemaSaveMode.RECREATE_SCHEMA) {
+ isNewTableCreated = true;
+ }
+ }
+
+ // Schema Coherence Validation: Only applies if the table already existed (not newly
+ // created)
+ // and we aren't ignoring schema changes or recreating schema.
+ if (!isNewTableCreated && tableState == TableState.EXIST_WRONG_SCHEMA) {
+ if (schemaSaveMode == SchemaSaveMode.CREATE_SCHEMA_WHEN_NOT_EXIST
+ || schemaSaveMode == SchemaSaveMode.ERROR_WHEN_SCHEMA_NOT_EXIST) {
+ return ExpectedOutcome.THROWS_COHERENCE_MISMATCH; // Schema Coherence mismatch error
+ }
+ }
+
+ // Phase 2: Data Stage behavior
+ if (!isNewTableCreated && tableState != TableState.NOT_EXIST) {
+ if (dataSaveMode == DataSaveMode.ERROR_WHEN_DATA_EXISTS) {
+ return ExpectedOutcome.THROWS_DATA_ALREADY_EXISTS; // Fatal Data Error
+ }
+ }
+
+ return ExpectedOutcome.SUCCESS;
+ }
+
+ @ParameterizedTest(
+ name = "Case: Schema={0}, Data={1}, TableState={2}, Batch={3} -> Expected={4}")
+ @MethodSource("saveModeMatrixProvider")
+ public void testCombinatorialSaveModeMatrix(
+ SchemaSaveMode schemaSaveMode,
+ DataSaveMode dataSaveMode,
+ TableState tableState,
+ boolean isBatch,
+ ExpectedOutcome expected)
+ throws Exception {
+
+ TablePath tablePath = TablePath.of("test-dataset", "matrix-table");
+ TableId tableId = TableId.of("test-dataset", "matrix-table");
+ CatalogTable catalogTable = createMockCatalogTable(tablePath);
+
+ // Stub Dataset exist checks
+ when(mockBigQuery.getDataset("test-dataset")).thenReturn(mock(Dataset.class));
+
+ // Mock target table behavior based on the testing state
+ if (tableState == TableState.NOT_EXIST) {
+ if (schemaSaveMode == SchemaSaveMode.CREATE_SCHEMA_WHEN_NOT_EXIST) {
+ Table mockTable = mock(Table.class);
+ TableDefinition mockDef = mock(TableDefinition.class);
+ when(mockTable.getDefinition()).thenReturn(mockDef);
+ when(mockTable.getNumRows()).thenReturn(java.math.BigInteger.ZERO);
+ com.google.cloud.bigquery.Schema correctSchema =
+ com.google.cloud.bigquery.Schema.of(
+ Field.of("id", StandardSQLTypeName.INT64));
+ when(mockDef.getSchema()).thenReturn(correctSchema);
+
+ when(mockBigQuery.getTable(tableId)).thenReturn(null).thenReturn(mockTable);
+ } else {
+ when(mockBigQuery.getTable(tableId)).thenReturn(null);
+ }
+ } else {
+ Table mockTable = mock(Table.class);
+ TableDefinition mockDef = mock(TableDefinition.class);
+ when(mockTable.getDefinition()).thenReturn(mockDef);
+ when(mockTable.getNumRows())
+ .thenReturn(java.math.BigInteger.valueOf(100)); // Has existing rows
+
+ if (tableState == TableState.EXIST_CORRECT_SCHEMA) {
+ com.google.cloud.bigquery.Schema correctSchema =
+ com.google.cloud.bigquery.Schema.of(
+ Field.of("id", StandardSQLTypeName.INT64));
+ when(mockDef.getSchema()).thenReturn(correctSchema);
+ } else {
+ com.google.cloud.bigquery.Schema wrongSchema =
+ com.google.cloud.bigquery.Schema.of(
+ Field.of("id", StandardSQLTypeName.STRING));
+ when(mockDef.getSchema()).thenReturn(wrongSchema);
+ }
+
+ // Sequential stubbing: first getTable returns existing table, second returns null
+ // (after delete/recreate)
+ if (schemaSaveMode == SchemaSaveMode.RECREATE_SCHEMA) {
+ when(mockBigQuery.getTable(tableId)).thenReturn(mockTable).thenReturn(null);
+ } else {
+ when(mockBigQuery.getTable(tableId)).thenReturn(mockTable);
+ }
+ }
+
+ // Mock deletion success
+ when(mockBigQuery.delete(tableId)).thenReturn(true);
+
+ // Dynamically construct a new BigQueryCatalog for each test run using isBatch
+ Map<String, Object> localConfigMap = new HashMap<>();
+ localConfigMap.put("project_id", "test-project");
+ localConfigMap.put("dataset_id", "test-dataset");
+ localConfigMap.put("write_mode", isBatch ? "batch" : "streaming");
+ ReadonlyConfig localConfig = ReadonlyConfig.fromMap(localConfigMap);
+
+ BigQueryCatalog localCatalog = new BigQueryCatalog("test-catalog", localConfig);
+
+ // Inject the mocked BigQuery client into this localized catalog using reflection
+ java.lang.reflect.Field clientField = BigQueryCatalog.class.getDeclaredField("bigquery");
+ clientField.setAccessible(true);
+ clientField.set(localCatalog, mockBigQuery);
+
+ BigQuerySaveModeHandler handler =
+ new BigQuerySaveModeHandler(
+ schemaSaveMode,
+ dataSaveMode,
+ localCatalog,
+ tablePath,
+ catalogTable,
+ "SELECT 1");
+
+ // Run assertions according to expected outcomes
+ if (expected == ExpectedOutcome.THROWS_TABLE_NOT_EXIST) {
+ assertThrows(
+ SeaTunnelRuntimeException.class,
+ () -> {
+ handler.handleSchemaSaveMode();
+ });
+ } else if (expected == ExpectedOutcome.THROWS_COHERENCE_MISMATCH) {
+ assertThrows(
+ CatalogException.class,
+ () -> {
+ handler.handleSchemaSaveMode();
+ });
+ } else if (expected == ExpectedOutcome.THROWS_DATA_ALREADY_EXISTS) {
+ assertThrows(
+ SeaTunnelRuntimeException.class,
+ () -> {
+ handler.handleSchemaSaveMode();
+ handler.handleDataSaveMode();
+ });
+ } else {
+ // SUCCESS
+ assertDoesNotThrow(
+ () -> {
+ handler.handleSchemaSaveMode();
+ handler.handleDataSaveMode();
+ });
+ }
+ }
+
+ private CatalogTable createMockCatalogTable(TablePath tablePath) {
+ TableSchema.Builder builder = TableSchema.builder();
+ builder.column(PhysicalColumn.of("id", BasicType.INT_TYPE, 0, false, null, "Primary key"));
+ builder.primaryKey(
+ org.apache.seatunnel.api.table.catalog.PrimaryKey.of(
+ "id_pk", Collections.singletonList("id")));
+ return CatalogTable.of(
+ TableIdentifier.of("test-catalog", tablePath),
+ builder.build(),
+ Collections.emptyMap(),
+ Collections.emptyList(),
+ "BigQuery Catalog Test Table");
+ }
+}
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkBatchWriterTest.java b/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkBatchWriterTest.java
index a3749abb34..fd772273b1 100644
--- a/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkBatchWriterTest.java
+++ b/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkBatchWriterTest.java
@@ -123,9 +123,28 @@ class BigQuerySinkBatchWriterTest {
assertSame(streamWriter, sinkWriter.streamWriter);
}
+ @Test
+ void testCloseDoesNotFlushUncheckpointedRowsInBatchMode() {
+ TestingBigQueryWriter streamWriter =
+ new TestingBigQueryWriter(
+ ApiFutures.immediateFuture(AppendRowsResponse.newBuilder().build()));
+ BigQuerySinkBatchWriter sinkWriter = createSinkWriter(streamWriter);
+ sinkWriter.buffer.put("row-1");
+ sinkWriter.buffer.put("row-2");
+
+ sinkWriter.close();
+
+ // In batch mode (flushOnClose == false), close() must NOT append un-checkpointed rows
+ // to preserve 2PC stream offset consistency during state recovery.
+ assertEquals(0, streamWriter.appendCount);
+ assertEquals(0, streamWriter.successfulRowCount);
+ assertEquals(1, streamWriter.closeCount);
+ assertEquals(2, sinkWriter.buffer.length());
+ }
+
private static BigQuerySinkBatchWriter createSinkWriter(BigQueryWriter streamWriter) {
return new BigQuerySinkBatchWriter(
- ReadonlyConfig.fromMap(Collections.emptyMap()), streamWriter, null, null);
+ ReadonlyConfig.fromMap(Collections.emptyMap()), streamWriter, null);
}
private static AppendRowsResponse errorResponse(Code code) {
diff --git a/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkFactoryTest.java b/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkFactoryTest.java
index 8c8ba7da86..28ce38d6eb 100644
--- a/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkFactoryTest.java
+++ b/seatunnel-connectors-v2/connector-bigquery/src/test/java/org/apache/seatunnel/connectors/bigquery/sink/BigQuerySinkFactoryTest.java
@@ -52,4 +52,26 @@ class BigQuerySinkFactoryTest {
ConfigValidator.of(ReadonlyConfig.fromConfig(config))
.validate(factory.optionRule()));
}
+
+ @Test
+ void testUniverseDomainConfigurationParsing() {
+ Config config =
+ ConfigFactory.parseString(
+ BigQuerySinkOptions.PROJECT_ID.key()
+ + " = \"test-project\"\n"
+ + BigQuerySinkOptions.DATASET_ID.key()
+ + " = \"test_dataset\"\n"
+ + BigQuerySinkOptions.TABLE_ID.key()
+ + " = \"test_table\"\n"
+ + BigQuerySinkOptions.UNIVERSE_DOMAIN.key()
+ + " = \"s3nsapis.fr\"\n");
+
+ BigQuerySinkFactory factory = new BigQuerySinkFactory();
+ ReadonlyConfig readonlyConfig = ReadonlyConfig.fromConfig(config);
+
+ assertDoesNotThrow(() -> ConfigValidator.of(readonlyConfig).validate(factory.optionRule()));
+
+ org.junit.jupiter.api.Assertions.assertEquals(
+ "s3nsapis.fr", readonlyConfig.get(BigQuerySinkOptions.UNIVERSE_DOMAIN));
+ }
}