You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "lordcheng10 (via GitHub)" <gi...@apache.org> on 2023/06/01 07:51:55 UTC
[GitHub] [inlong] lordcheng10 opened a new pull request, #8141: [INLONG-8140][Sort] Support data inference schema change type
lordcheng10 opened a new pull request, #8141:
URL: https://github.com/apache/inlong/pull/8141
### Prepare a Pull Request
- Fixes https://github.com/apache/inlong/issues/8140
### Motivation
Support data inference schema change type.
Iceberg connector supports data inference schema change type,
The purpose of this PR is to extract some public interfaces, which can be used when other connectors implement data inference schema change type;
### Modifications
Iceberg connector supports data inference schema change type,
The purpose of this PR is to extract some public interfaces, which can be used when other connectors implement data inference schema change type;
### Verifying this change
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [ ] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1217558819
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/pom.xml:
##########
@@ -88,6 +88,12 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>sort-common</artifactId>
+ <version>1.8.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
Review Comment:
> Fixed
@lordcheng10 I just use it as a example by relocation of sort-connector-base. you should shade sort-common
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1217621006
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/pom.xml:
##########
@@ -143,8 +143,8 @@
<shadedPattern>org.apache.inlong.sort.iceberg.shaded.com.amazonaws</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.inlong.sort.base</pattern>
- <shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort.base</shadedPattern>
+ <pattern>org.apache.inlong.sort</pattern>
+ <shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort</shadedPattern>
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1217391231
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHandle.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+
+import java.util.List;
+import java.util.Map;
+
+public interface SchemaChangeHandle {
+
+ void process(byte[] originData, JsonNode data);
Review Comment:
Fixed
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Iceberg schema change helper
+ * */
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeHelper.class);
+ private transient Catalog catalog;
+
+ private transient SupportsNamespaces asNamespaceCatalog;
+
+ public IcebergSchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper,
+ Catalog catalog,
+ SupportsNamespaces asNamespaceCatalog) {
+ super(dynamicSchemaFormat, schemaChange, policyMap, databasePattern,
+ tablePattern, exceptionPolicy, metricData, dirtySinkHelper);
+ this.catalog = catalog;
+ this.asNamespaceCatalog = asNamespaceCatalog;
+ }
+ @Override
+ public void doAlterOperation(String database, String table, byte[] originData, String originSchema, JsonNode data,
+ Map<SchemaChangeType, List<AlterColumn>> typeMap) {
+ for (Map.Entry<SchemaChangeType, List<AlterColumn>> kv : typeMap.entrySet()) {
+ SchemaChangePolicy policy = policyMap.get(kv.getKey());
+ doSchemaChangeBase(kv.getKey(), policy, originSchema);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ switch (kv.getKey()) {
+ case ADD_COLUMN:
+ doAddColumn(kv.getValue(), TableIdentifier.of(database, table));
+ break;
+ case DROP_COLUMN:
+ doDropColumn(kv.getKey(), originSchema);
+ break;
+ case RENAME_COLUMN:
+ doRenameColumn(kv.getKey(), originSchema);
+ break;
+ case CHANGE_COLUMN_TYPE:
+ doChangeColumnType(kv.getKey(), originSchema);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1217402955
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/DynamicSchemaHandleOperator.java:
##########
@@ -434,33 +450,29 @@ private void handleSchemaInfoEvent(TableIdentifier tableId, Schema schema) {
// ================================ All coordinator handle method ==============================================
private void handleTableCreateEventFromOperator(TableIdentifier tableId, Schema schema) {
- if (!catalog.tableExists(tableId)) {
- if (asNamespaceCatalog != null && !asNamespaceCatalog.namespaceExists(tableId.namespace())) {
- try {
- asNamespaceCatalog.createNamespace(tableId.namespace());
- LOG.info("Auto create Database({}) in Catalog({}).", tableId.namespace(), catalog.name());
- } catch (AlreadyExistsException e) {
- LOG.warn("Database({}) already exist in Catalog({})!", tableId.namespace(), catalog.name());
- }
- }
- ImmutableMap.Builder<String, String> properties = ImmutableMap.builder();
- properties.put("format-version", "2");
- properties.put("write.upsert.enabled", "true");
- properties.put("write.metadata.metrics.default", "full");
- // for hive visible
- properties.put("engine.hive.enabled", "true");
- try {
- catalog.createTable(tableId, schema, PartitionSpec.unpartitioned(), properties.build());
- LOG.info("Auto create Table({}) in Database({}) in Catalog({})!",
- tableId.name(), tableId.namespace(), catalog.name());
- } catch (AlreadyExistsException e) {
- LOG.warn("Table({}) already exist in Database({}) in Catalog({})!",
- tableId.name(), tableId.namespace(), catalog.name());
- }
- }
+ IcebergSchemaChangeUtils.createTable(catalog, tableId, asNamespaceCatalog, schema);
handleSchemaInfoEvent(tableId, catalog.loadTable(tableId).schema());
}
+ @VisibleForTesting
+ public static Map<String, ColumnSchema> extractColumnSchema(Schema schema) {
+ Map<String, ColumnSchema> columnSchemaMap = new LinkedHashMap<>();
+ List<Types.NestedField> nestedFieldList = schema.columns();
+ int n = nestedFieldList.size();
+ for (int i = 0; i < n; i++) {
+ Types.NestedField nestedField = nestedFieldList.get(i);
+ ColumnSchema columnSchema = new ColumnSchema();
+ columnSchema.setName(nestedField.name());
+ columnSchema.setType(FlinkSchemaUtil.convert(nestedField.type()));
+ columnSchema.setNullable(nestedField.isOptional());
+ columnSchema.setComment(nestedField.doc());
+ columnSchema.setPosition(i == 0 ? TableChange.ColumnPosition.first()
+ : TableChange.ColumnPosition.after(nestedFieldList.get(n - 1).name()));
Review Comment:
You are right! Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1213936565
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/util/SchemaChangeUtils.java:
##########
@@ -23,25 +23,31 @@
import org.apache.inlong.sort.protocol.ddl.operations.Operation;
import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.ColumnSchema;
+import org.apache.inlong.sort.schema.TableChange;
import com.google.common.base.Preconditions;
+import com.google.common.collect.Sets;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import java.util.StringJoiner;
+import java.util.stream.Collectors;
/**
* Schema-change Utils
*/
-public final class SchemaChangeUtils {
+public class SchemaChangeUtils {
Review Comment:
A class modified with final cannot be inherited.
IcebergSchemaChangeUtils inherits from SchemaChangeUtils. @yunqingmoswu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214163757
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/pom.xml:
##########
@@ -88,6 +88,12 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>sort-common</artifactId>
+ <version>1.8.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
Review Comment:
I suggest add relocation if you import this dependency.
like sort-connector-base
```
<relocation>
<pattern>org.apache.inlong.sort.base</pattern>
<shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort.base</shadedPattern>
</relocation>
```
Because some not changed connector won't be effected when modify sort-common and connector also dependency sort-common.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1213963793
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/schema/ColumnSchema.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.schema;
+
+import lombok.Data;
+import org.apache.flink.table.types.logical.LogicalType;
+
+@Data
+public class ColumnSchema {
+
+ private String name;
+ private LogicalType type;
+ private boolean isNullable;
+ private String doc;
Review Comment:
doc rename to comment.
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/schema/ColumnSchema.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.schema;
+
+import lombok.Data;
+import org.apache.flink.table.types.logical.LogicalType;
+
+@Data
+public class ColumnSchema {
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214172869
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/IcebergSchemaChangeUtils.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.sink.multiple;
+
+import org.apache.inlong.sort.iceberg.FlinkTypeToType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.UpdateSchema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+
+public class IcebergSchemaChangeUtils extends SchemaChangeUtils {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeUtils.class);
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ public static void createTable(Catalog catalog, TableIdentifier tableId, SupportsNamespaces asNamespaceCatalog,
+ Schema schema) {
+ if (!catalog.tableExists(tableId)) {
Review Comment:
I can modify this method name, for example: createTableIfNotExit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214121370
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHandle.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+
+import java.util.List;
+import java.util.Map;
+
+public interface SchemaChangeHandle {
+
+ void process(byte[] originData, JsonNode data);
+
+ void handleAlterOperation(String database, String table, byte[] originData, String originSchema,
+ JsonNode data, AlterOperation operation);
+
+ void doAlterOperation(String database, String table, byte[] originData, String originSchema, JsonNode data,
+ Map<SchemaChangeType, List<AlterColumn>> typeMap);
+ String doAddColumn(SchemaChangeType type, String originSchema);
+
+ String doChangeColumnType(SchemaChangeType type, String originSchema);
+
+ String doRenameColumn(SchemaChangeType type, String originSchema);
Review Comment:
The return value is not string inside the method, what's it for ? maybe void if string is not used
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHandle.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+
+import java.util.List;
+import java.util.Map;
+
+public interface SchemaChangeHandle {
+
+ void process(byte[] originData, JsonNode data);
Review Comment:
please add java doc for interface method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214151382
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Schema change helper
+ * */
+public abstract class SchemaChangeHelper implements SchemaChangeHandle {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SchemaChangeHelper.class);
+ private final boolean schemaChange;
+ protected final Map<SchemaChangeType, SchemaChangePolicy> policyMap;
+ protected final JsonDynamicSchemaFormat dynamicSchemaFormat;
+ private final String databasePattern;
+ private final String tablePattern;
+ protected final SchemaUpdateExceptionPolicy exceptionPolicy;
+ private final SinkTableMetricData metricData;
+ private final DirtySinkHelper<Object> dirtySinkHelper;
+
+ public SchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper) {
+ this.dynamicSchemaFormat = Preconditions.checkNotNull(dynamicSchemaFormat, "dynamicSchemaFormat is null");
+ this.schemaChange = schemaChange;
+ this.policyMap = policyMap;
+ this.databasePattern = databasePattern;
+ this.tablePattern = tablePattern;
+ this.exceptionPolicy = exceptionPolicy;
+ this.metricData = metricData;
+ this.dirtySinkHelper = dirtySinkHelper;
+ }
+
+ public void process(byte[] originData, JsonNode data) {
+ // 1. Extract the schema change type from the data;
+ if (!schemaChange) {
+ return;
+ }
+ String database;
+ String table;
+ try {
+ database = dynamicSchemaFormat.parse(data, databasePattern);
+ table = dynamicSchemaFormat.parse(data, tablePattern);
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Parse database, table from origin data failed, origin data: %s",
+ new String(originData)),
+ e);
+ }
+ LOGGER.warn("Parse database, table from origin data failed, origin data: {}", new String(originData), e);
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.LOG_WITH_IGNORE) {
+ dirtySinkHelper.invoke(new String(originData), DirtyType.JSON_PROCESS_ERROR, e);
+ }
+ if (metricData != null) {
+ metricData.invokeDirty(1, originData.length);
+ }
+ return;
+ }
+ Operation operation;
+ try {
+ JsonNode operationNode = Preconditions.checkNotNull(data.get("operation"),
+ "Operation node is null");
+ operation = Preconditions.checkNotNull(
+ dynamicSchemaFormat.objectMapper.convertValue(operationNode, new TypeReference<Operation>() {
+ }), "Operation is null");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract Operation from origin data failed,origin data: %s", data), e);
+ }
+ LOGGER.warn("Extract Operation from origin data failed,origin data: {}", data, e);
+ handleDirtyData(data, originData, database, table, DirtyType.JSON_PROCESS_ERROR, e);
+ return;
+ }
+ String originSchema = dynamicSchemaFormat.extractDDL(data);
+ SchemaChangeType type = SchemaChangeUtils.extractSchemaChangeType(operation);
+ if (type == null) {
+ LOGGER.warn("Unsupported for schema-change: {}", originSchema);
+ return;
+ }
+
+ // 2. Apply schema change;
+ switch (type) {
+ case ALTER:
+ handleAlterOperation(database, table, originData, originSchema, data, (AlterOperation) operation);
+ break;
+ case CREATE_TABLE:
+ doCreateTable(originData, database, table, type, originSchema, data, (CreateTableOperation) operation);
+ break;
+ case DROP_TABLE:
+ doDropTable(type, originSchema);
+ break;
+ case RENAME_TABLE:
+ doRenameTable(type, originSchema);
+ break;
+ case TRUNCATE_TABLE:
+ doTruncateTable(type, originSchema);
+ break;
+ default:
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ }
+ }
+
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ doAlterOperation(database, table, originData, originSchema, data, typeMap);
+ }
+ }
+
+ @Override
+ public String doAddColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doRenameColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doDropColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
Review Comment:
return null? Why not to set method return type to void.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1213965736
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214112864
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/IcebergSchemaChangeUtils.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.sink.multiple;
+
+import org.apache.inlong.sort.iceberg.FlinkTypeToType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.UpdateSchema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+
+public class IcebergSchemaChangeUtils extends SchemaChangeUtils {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeUtils.class);
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ public static void createTable(Catalog catalog, TableIdentifier tableId, SupportsNamespaces asNamespaceCatalog,
+ Schema schema) {
+ if (!catalog.tableExists(tableId)) {
Review Comment:
Maybe the method name should be consistent with the method inplement. Here createtable method do a if-else before actually create table. I guess the if-else should be outside of the method and functioned by the caller of the method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214186268
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/DynamicSchemaHandleOperator.java:
##########
@@ -434,33 +450,29 @@ private void handleSchemaInfoEvent(TableIdentifier tableId, Schema schema) {
// ================================ All coordinator handle method ==============================================
private void handleTableCreateEventFromOperator(TableIdentifier tableId, Schema schema) {
- if (!catalog.tableExists(tableId)) {
- if (asNamespaceCatalog != null && !asNamespaceCatalog.namespaceExists(tableId.namespace())) {
- try {
- asNamespaceCatalog.createNamespace(tableId.namespace());
- LOG.info("Auto create Database({}) in Catalog({}).", tableId.namespace(), catalog.name());
- } catch (AlreadyExistsException e) {
- LOG.warn("Database({}) already exist in Catalog({})!", tableId.namespace(), catalog.name());
- }
- }
- ImmutableMap.Builder<String, String> properties = ImmutableMap.builder();
- properties.put("format-version", "2");
- properties.put("write.upsert.enabled", "true");
- properties.put("write.metadata.metrics.default", "full");
- // for hive visible
- properties.put("engine.hive.enabled", "true");
- try {
- catalog.createTable(tableId, schema, PartitionSpec.unpartitioned(), properties.build());
- LOG.info("Auto create Table({}) in Database({}) in Catalog({})!",
- tableId.name(), tableId.namespace(), catalog.name());
- } catch (AlreadyExistsException e) {
- LOG.warn("Table({}) already exist in Database({}) in Catalog({})!",
- tableId.name(), tableId.namespace(), catalog.name());
- }
- }
+ IcebergSchemaChangeUtils.createTable(catalog, tableId, asNamespaceCatalog, schema);
handleSchemaInfoEvent(tableId, catalog.loadTable(tableId).schema());
}
+ @VisibleForTesting
+ public static Map<String, ColumnSchema> extractColumnSchema(Schema schema) {
+ Map<String, ColumnSchema> columnSchemaMap = new LinkedHashMap<>();
+ List<Types.NestedField> nestedFieldList = schema.columns();
+ int n = nestedFieldList.size();
+ for (int i = 0; i < n; i++) {
+ Types.NestedField nestedField = nestedFieldList.get(i);
+ ColumnSchema columnSchema = new ColumnSchema();
+ columnSchema.setName(nestedField.name());
+ columnSchema.setType(FlinkSchemaUtil.convert(nestedField.type()));
+ columnSchema.setNullable(nestedField.isOptional());
+ columnSchema.setComment(nestedField.doc());
+ columnSchema.setPosition(i == 0 ? TableChange.ColumnPosition.first()
+ : TableChange.ColumnPosition.after(nestedFieldList.get(n - 1).name()));
Review Comment:
`TableChange.ColumnPosition.after(nestedFieldList.get(n - 1).name())` is right ?
nestedFieldList.get(n - 1) maybe nestedFieldList.get(i - 1) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on PR #8141:
URL: https://github.com/apache/inlong/pull/8141#issuecomment-1575980873
@gong @EMsnap PTAL,thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214174208
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Schema change helper
+ * */
+public abstract class SchemaChangeHelper implements SchemaChangeHandle {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SchemaChangeHelper.class);
+ private final boolean schemaChange;
+ protected final Map<SchemaChangeType, SchemaChangePolicy> policyMap;
+ protected final JsonDynamicSchemaFormat dynamicSchemaFormat;
+ private final String databasePattern;
+ private final String tablePattern;
+ protected final SchemaUpdateExceptionPolicy exceptionPolicy;
+ private final SinkTableMetricData metricData;
+ private final DirtySinkHelper<Object> dirtySinkHelper;
+
+ public SchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper) {
+ this.dynamicSchemaFormat = Preconditions.checkNotNull(dynamicSchemaFormat, "dynamicSchemaFormat is null");
+ this.schemaChange = schemaChange;
+ this.policyMap = policyMap;
+ this.databasePattern = databasePattern;
+ this.tablePattern = tablePattern;
+ this.exceptionPolicy = exceptionPolicy;
+ this.metricData = metricData;
+ this.dirtySinkHelper = dirtySinkHelper;
+ }
+
+ public void process(byte[] originData, JsonNode data) {
+ // 1. Extract the schema change type from the data;
+ if (!schemaChange) {
+ return;
+ }
+ String database;
+ String table;
+ try {
+ database = dynamicSchemaFormat.parse(data, databasePattern);
+ table = dynamicSchemaFormat.parse(data, tablePattern);
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Parse database, table from origin data failed, origin data: %s",
+ new String(originData)),
+ e);
+ }
+ LOGGER.warn("Parse database, table from origin data failed, origin data: {}", new String(originData), e);
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.LOG_WITH_IGNORE) {
+ dirtySinkHelper.invoke(new String(originData), DirtyType.JSON_PROCESS_ERROR, e);
+ }
+ if (metricData != null) {
+ metricData.invokeDirty(1, originData.length);
+ }
+ return;
+ }
+ Operation operation;
+ try {
+ JsonNode operationNode = Preconditions.checkNotNull(data.get("operation"),
+ "Operation node is null");
+ operation = Preconditions.checkNotNull(
+ dynamicSchemaFormat.objectMapper.convertValue(operationNode, new TypeReference<Operation>() {
+ }), "Operation is null");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract Operation from origin data failed,origin data: %s", data), e);
+ }
+ LOGGER.warn("Extract Operation from origin data failed,origin data: {}", data, e);
+ handleDirtyData(data, originData, database, table, DirtyType.JSON_PROCESS_ERROR, e);
+ return;
+ }
+ String originSchema = dynamicSchemaFormat.extractDDL(data);
+ SchemaChangeType type = SchemaChangeUtils.extractSchemaChangeType(operation);
+ if (type == null) {
+ LOGGER.warn("Unsupported for schema-change: {}", originSchema);
+ return;
+ }
+
+ // 2. Apply schema change;
+ switch (type) {
+ case ALTER:
+ handleAlterOperation(database, table, originData, originSchema, data, (AlterOperation) operation);
+ break;
+ case CREATE_TABLE:
+ doCreateTable(originData, database, table, type, originSchema, data, (CreateTableOperation) operation);
+ break;
+ case DROP_TABLE:
+ doDropTable(type, originSchema);
+ break;
+ case RENAME_TABLE:
+ doRenameTable(type, originSchema);
+ break;
+ case TRUNCATE_TABLE:
+ doTruncateTable(type, originSchema);
+ break;
+ default:
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ }
+ }
+
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ doAlterOperation(database, table, originData, originSchema, data, typeMap);
+ }
+ }
+
+ @Override
+ public String doAddColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doRenameColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doDropColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
Review Comment:
Doris connector may use the return value of this method
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Schema change helper
+ * */
+public abstract class SchemaChangeHelper implements SchemaChangeHandle {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SchemaChangeHelper.class);
+ private final boolean schemaChange;
+ protected final Map<SchemaChangeType, SchemaChangePolicy> policyMap;
+ protected final JsonDynamicSchemaFormat dynamicSchemaFormat;
+ private final String databasePattern;
+ private final String tablePattern;
+ protected final SchemaUpdateExceptionPolicy exceptionPolicy;
+ private final SinkTableMetricData metricData;
+ private final DirtySinkHelper<Object> dirtySinkHelper;
+
+ public SchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper) {
+ this.dynamicSchemaFormat = Preconditions.checkNotNull(dynamicSchemaFormat, "dynamicSchemaFormat is null");
+ this.schemaChange = schemaChange;
+ this.policyMap = policyMap;
+ this.databasePattern = databasePattern;
+ this.tablePattern = tablePattern;
+ this.exceptionPolicy = exceptionPolicy;
+ this.metricData = metricData;
+ this.dirtySinkHelper = dirtySinkHelper;
+ }
+
+ public void process(byte[] originData, JsonNode data) {
+ // 1. Extract the schema change type from the data;
+ if (!schemaChange) {
+ return;
+ }
+ String database;
+ String table;
+ try {
+ database = dynamicSchemaFormat.parse(data, databasePattern);
+ table = dynamicSchemaFormat.parse(data, tablePattern);
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Parse database, table from origin data failed, origin data: %s",
+ new String(originData)),
+ e);
+ }
+ LOGGER.warn("Parse database, table from origin data failed, origin data: {}", new String(originData), e);
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.LOG_WITH_IGNORE) {
+ dirtySinkHelper.invoke(new String(originData), DirtyType.JSON_PROCESS_ERROR, e);
+ }
+ if (metricData != null) {
+ metricData.invokeDirty(1, originData.length);
+ }
+ return;
+ }
+ Operation operation;
+ try {
+ JsonNode operationNode = Preconditions.checkNotNull(data.get("operation"),
+ "Operation node is null");
+ operation = Preconditions.checkNotNull(
+ dynamicSchemaFormat.objectMapper.convertValue(operationNode, new TypeReference<Operation>() {
+ }), "Operation is null");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract Operation from origin data failed,origin data: %s", data), e);
+ }
+ LOGGER.warn("Extract Operation from origin data failed,origin data: {}", data, e);
+ handleDirtyData(data, originData, database, table, DirtyType.JSON_PROCESS_ERROR, e);
+ return;
+ }
+ String originSchema = dynamicSchemaFormat.extractDDL(data);
+ SchemaChangeType type = SchemaChangeUtils.extractSchemaChangeType(operation);
+ if (type == null) {
+ LOGGER.warn("Unsupported for schema-change: {}", originSchema);
+ return;
+ }
+
+ // 2. Apply schema change;
+ switch (type) {
+ case ALTER:
+ handleAlterOperation(database, table, originData, originSchema, data, (AlterOperation) operation);
+ break;
+ case CREATE_TABLE:
+ doCreateTable(originData, database, table, type, originSchema, data, (CreateTableOperation) operation);
+ break;
+ case DROP_TABLE:
+ doDropTable(type, originSchema);
+ break;
+ case RENAME_TABLE:
+ doRenameTable(type, originSchema);
+ break;
+ case TRUNCATE_TABLE:
+ doTruncateTable(type, originSchema);
+ break;
+ default:
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ }
+ }
+
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ doAlterOperation(database, table, originData, originSchema, data, typeMap);
+ }
+ }
+
+ @Override
+ public String doAddColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doRenameColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doDropColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214172869
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/IcebergSchemaChangeUtils.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.sink.multiple;
+
+import org.apache.inlong.sort.iceberg.FlinkTypeToType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.UpdateSchema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+
+public class IcebergSchemaChangeUtils extends SchemaChangeUtils {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeUtils.class);
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ public static void createTable(Catalog catalog, TableIdentifier tableId, SupportsNamespaces asNamespaceCatalog,
+ Schema schema) {
+ if (!catalog.tableExists(tableId)) {
Review Comment:
I can modify this method name, for example: createTableIfNotExit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214163757
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/pom.xml:
##########
@@ -88,6 +88,12 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>sort-common</artifactId>
+ <version>1.8.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
Review Comment:
I suggest add relocation if you import this dependency.
like sort-connector-base
```
<relocation>
<pattern>org.apache.inlong.sort.base</pattern>
<shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort.base</shadedPattern>
</relocation>
```
Because some not changed connector won't be effected when modify sort-common and connector also dependency sort-common.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214150629
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Schema change helper
+ * */
+public abstract class SchemaChangeHelper implements SchemaChangeHandle {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SchemaChangeHelper.class);
+ private final boolean schemaChange;
+ protected final Map<SchemaChangeType, SchemaChangePolicy> policyMap;
+ protected final JsonDynamicSchemaFormat dynamicSchemaFormat;
+ private final String databasePattern;
+ private final String tablePattern;
+ protected final SchemaUpdateExceptionPolicy exceptionPolicy;
+ private final SinkTableMetricData metricData;
+ private final DirtySinkHelper<Object> dirtySinkHelper;
+
+ public SchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper) {
+ this.dynamicSchemaFormat = Preconditions.checkNotNull(dynamicSchemaFormat, "dynamicSchemaFormat is null");
+ this.schemaChange = schemaChange;
+ this.policyMap = policyMap;
+ this.databasePattern = databasePattern;
+ this.tablePattern = tablePattern;
+ this.exceptionPolicy = exceptionPolicy;
+ this.metricData = metricData;
+ this.dirtySinkHelper = dirtySinkHelper;
+ }
+
+ public void process(byte[] originData, JsonNode data) {
+ // 1. Extract the schema change type from the data;
+ if (!schemaChange) {
+ return;
+ }
+ String database;
+ String table;
+ try {
+ database = dynamicSchemaFormat.parse(data, databasePattern);
+ table = dynamicSchemaFormat.parse(data, tablePattern);
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Parse database, table from origin data failed, origin data: %s",
+ new String(originData)),
+ e);
+ }
+ LOGGER.warn("Parse database, table from origin data failed, origin data: {}", new String(originData), e);
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.LOG_WITH_IGNORE) {
+ dirtySinkHelper.invoke(new String(originData), DirtyType.JSON_PROCESS_ERROR, e);
+ }
+ if (metricData != null) {
+ metricData.invokeDirty(1, originData.length);
+ }
+ return;
+ }
+ Operation operation;
+ try {
+ JsonNode operationNode = Preconditions.checkNotNull(data.get("operation"),
+ "Operation node is null");
+ operation = Preconditions.checkNotNull(
+ dynamicSchemaFormat.objectMapper.convertValue(operationNode, new TypeReference<Operation>() {
+ }), "Operation is null");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract Operation from origin data failed,origin data: %s", data), e);
+ }
+ LOGGER.warn("Extract Operation from origin data failed,origin data: {}", data, e);
+ handleDirtyData(data, originData, database, table, DirtyType.JSON_PROCESS_ERROR, e);
+ return;
+ }
+ String originSchema = dynamicSchemaFormat.extractDDL(data);
+ SchemaChangeType type = SchemaChangeUtils.extractSchemaChangeType(operation);
+ if (type == null) {
+ LOGGER.warn("Unsupported for schema-change: {}", originSchema);
+ return;
+ }
+
+ // 2. Apply schema change;
+ switch (type) {
+ case ALTER:
+ handleAlterOperation(database, table, originData, originSchema, data, (AlterOperation) operation);
+ break;
+ case CREATE_TABLE:
+ doCreateTable(originData, database, table, type, originSchema, data, (CreateTableOperation) operation);
+ break;
+ case DROP_TABLE:
+ doDropTable(type, originSchema);
+ break;
+ case RENAME_TABLE:
+ doRenameTable(type, originSchema);
+ break;
+ case TRUNCATE_TABLE:
+ doTruncateTable(type, originSchema);
+ break;
+ default:
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ }
+ }
+
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ doAlterOperation(database, table, originData, originSchema, data, typeMap);
+ }
+ }
+
+ @Override
+ public String doAddColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
Review Comment:
return null? Why not to set method return type to void.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1217564063
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/pom.xml:
##########
@@ -143,8 +143,8 @@
<shadedPattern>org.apache.inlong.sort.iceberg.shaded.com.amazonaws</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.inlong.sort.base</pattern>
- <shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort.base</shadedPattern>
+ <pattern>org.apache.inlong.sort</pattern>
+ <shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort</shadedPattern>
Review Comment:
@lordcheng10` <pattern>org.apache.inlong.sort</pattern> `and `<pattern>org.apache.inlong.sort.base</pattern>` will be effected each other?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1217564063
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/pom.xml:
##########
@@ -143,8 +143,8 @@
<shadedPattern>org.apache.inlong.sort.iceberg.shaded.com.amazonaws</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.inlong.sort.base</pattern>
- <shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort.base</shadedPattern>
+ <pattern>org.apache.inlong.sort</pattern>
+ <shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort</shadedPattern>
Review Comment:
@lordcheng10 <pattern>org.apache.inlong.sort</pattern> and <pattern>org.apache.inlong.sort.base</pattern> will be effected each other?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1213963951
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeInterface.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+
+public interface SchemaChangeInterface {
Review Comment:
OK
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+
+public abstract class SchemaChangeHelper implements SchemaChangeInterface {
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214174054
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Schema change helper
+ * */
+public abstract class SchemaChangeHelper implements SchemaChangeHandle {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SchemaChangeHelper.class);
+ private final boolean schemaChange;
+ protected final Map<SchemaChangeType, SchemaChangePolicy> policyMap;
+ protected final JsonDynamicSchemaFormat dynamicSchemaFormat;
+ private final String databasePattern;
+ private final String tablePattern;
+ protected final SchemaUpdateExceptionPolicy exceptionPolicy;
+ private final SinkTableMetricData metricData;
+ private final DirtySinkHelper<Object> dirtySinkHelper;
+
+ public SchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper) {
+ this.dynamicSchemaFormat = Preconditions.checkNotNull(dynamicSchemaFormat, "dynamicSchemaFormat is null");
+ this.schemaChange = schemaChange;
+ this.policyMap = policyMap;
+ this.databasePattern = databasePattern;
+ this.tablePattern = tablePattern;
+ this.exceptionPolicy = exceptionPolicy;
+ this.metricData = metricData;
+ this.dirtySinkHelper = dirtySinkHelper;
+ }
+
+ public void process(byte[] originData, JsonNode data) {
+ // 1. Extract the schema change type from the data;
+ if (!schemaChange) {
+ return;
+ }
+ String database;
+ String table;
+ try {
+ database = dynamicSchemaFormat.parse(data, databasePattern);
+ table = dynamicSchemaFormat.parse(data, tablePattern);
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Parse database, table from origin data failed, origin data: %s",
+ new String(originData)),
+ e);
+ }
+ LOGGER.warn("Parse database, table from origin data failed, origin data: {}", new String(originData), e);
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.LOG_WITH_IGNORE) {
+ dirtySinkHelper.invoke(new String(originData), DirtyType.JSON_PROCESS_ERROR, e);
+ }
+ if (metricData != null) {
+ metricData.invokeDirty(1, originData.length);
+ }
+ return;
+ }
+ Operation operation;
+ try {
+ JsonNode operationNode = Preconditions.checkNotNull(data.get("operation"),
+ "Operation node is null");
+ operation = Preconditions.checkNotNull(
+ dynamicSchemaFormat.objectMapper.convertValue(operationNode, new TypeReference<Operation>() {
+ }), "Operation is null");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract Operation from origin data failed,origin data: %s", data), e);
+ }
+ LOGGER.warn("Extract Operation from origin data failed,origin data: {}", data, e);
+ handleDirtyData(data, originData, database, table, DirtyType.JSON_PROCESS_ERROR, e);
+ return;
+ }
+ String originSchema = dynamicSchemaFormat.extractDDL(data);
+ SchemaChangeType type = SchemaChangeUtils.extractSchemaChangeType(operation);
+ if (type == null) {
+ LOGGER.warn("Unsupported for schema-change: {}", originSchema);
+ return;
+ }
+
+ // 2. Apply schema change;
+ switch (type) {
+ case ALTER:
+ handleAlterOperation(database, table, originData, originSchema, data, (AlterOperation) operation);
+ break;
+ case CREATE_TABLE:
+ doCreateTable(originData, database, table, type, originSchema, data, (CreateTableOperation) operation);
+ break;
+ case DROP_TABLE:
+ doDropTable(type, originSchema);
+ break;
+ case RENAME_TABLE:
+ doRenameTable(type, originSchema);
+ break;
+ case TRUNCATE_TABLE:
+ doTruncateTable(type, originSchema);
+ break;
+ default:
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ }
+ }
+
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ doAlterOperation(database, table, originData, originSchema, data, typeMap);
+ }
+ }
+
+ @Override
+ public String doAddColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
Review Comment:
Doris connector may use the return value of this method
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Schema change helper
+ * */
+public abstract class SchemaChangeHelper implements SchemaChangeHandle {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SchemaChangeHelper.class);
+ private final boolean schemaChange;
+ protected final Map<SchemaChangeType, SchemaChangePolicy> policyMap;
+ protected final JsonDynamicSchemaFormat dynamicSchemaFormat;
+ private final String databasePattern;
+ private final String tablePattern;
+ protected final SchemaUpdateExceptionPolicy exceptionPolicy;
+ private final SinkTableMetricData metricData;
+ private final DirtySinkHelper<Object> dirtySinkHelper;
+
+ public SchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper) {
+ this.dynamicSchemaFormat = Preconditions.checkNotNull(dynamicSchemaFormat, "dynamicSchemaFormat is null");
+ this.schemaChange = schemaChange;
+ this.policyMap = policyMap;
+ this.databasePattern = databasePattern;
+ this.tablePattern = tablePattern;
+ this.exceptionPolicy = exceptionPolicy;
+ this.metricData = metricData;
+ this.dirtySinkHelper = dirtySinkHelper;
+ }
+
+ public void process(byte[] originData, JsonNode data) {
+ // 1. Extract the schema change type from the data;
+ if (!schemaChange) {
+ return;
+ }
+ String database;
+ String table;
+ try {
+ database = dynamicSchemaFormat.parse(data, databasePattern);
+ table = dynamicSchemaFormat.parse(data, tablePattern);
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Parse database, table from origin data failed, origin data: %s",
+ new String(originData)),
+ e);
+ }
+ LOGGER.warn("Parse database, table from origin data failed, origin data: {}", new String(originData), e);
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.LOG_WITH_IGNORE) {
+ dirtySinkHelper.invoke(new String(originData), DirtyType.JSON_PROCESS_ERROR, e);
+ }
+ if (metricData != null) {
+ metricData.invokeDirty(1, originData.length);
+ }
+ return;
+ }
+ Operation operation;
+ try {
+ JsonNode operationNode = Preconditions.checkNotNull(data.get("operation"),
+ "Operation node is null");
+ operation = Preconditions.checkNotNull(
+ dynamicSchemaFormat.objectMapper.convertValue(operationNode, new TypeReference<Operation>() {
+ }), "Operation is null");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract Operation from origin data failed,origin data: %s", data), e);
+ }
+ LOGGER.warn("Extract Operation from origin data failed,origin data: {}", data, e);
+ handleDirtyData(data, originData, database, table, DirtyType.JSON_PROCESS_ERROR, e);
+ return;
+ }
+ String originSchema = dynamicSchemaFormat.extractDDL(data);
+ SchemaChangeType type = SchemaChangeUtils.extractSchemaChangeType(operation);
+ if (type == null) {
+ LOGGER.warn("Unsupported for schema-change: {}", originSchema);
+ return;
+ }
+
+ // 2. Apply schema change;
+ switch (type) {
+ case ALTER:
+ handleAlterOperation(database, table, originData, originSchema, data, (AlterOperation) operation);
+ break;
+ case CREATE_TABLE:
+ doCreateTable(originData, database, table, type, originSchema, data, (CreateTableOperation) operation);
+ break;
+ case DROP_TABLE:
+ doDropTable(type, originSchema);
+ break;
+ case RENAME_TABLE:
+ doRenameTable(type, originSchema);
+ break;
+ case TRUNCATE_TABLE:
+ doTruncateTable(type, originSchema);
+ break;
+ default:
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ }
+ }
+
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ doAlterOperation(database, table, originData, originSchema, data, typeMap);
+ }
+ }
+
+ @Override
+ public String doAddColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214173640
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHandle.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+
+import java.util.List;
+import java.util.Map;
+
+public interface SchemaChangeHandle {
+
+ void process(byte[] originData, JsonNode data);
+
+ void handleAlterOperation(String database, String table, byte[] originData, String originSchema,
+ JsonNode data, AlterOperation operation);
+
+ void doAlterOperation(String database, String table, byte[] originData, String originSchema, JsonNode data,
+ Map<SchemaChangeType, List<AlterColumn>> typeMap);
+ String doAddColumn(SchemaChangeType type, String originSchema);
+
+ String doChangeColumnType(SchemaChangeType type, String originSchema);
+
+ String doRenameColumn(SchemaChangeType type, String originSchema);
Review Comment:
doris connector may use the return value of this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "yunqingmoswu (via GitHub)" <gi...@apache.org>.
yunqingmoswu commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1213891104
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/schema/ColumnSchema.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.schema;
+
+import lombok.Data;
+import org.apache.flink.table.types.logical.LogicalType;
+
+@Data
+public class ColumnSchema {
+
+ private String name;
+ private LogicalType type;
+ private boolean isNullable;
+ private String doc;
Review Comment:
What's the meaning of this?
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/schema/ColumnSchema.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.schema;
+
+import lombok.Data;
+import org.apache.flink.table.types.logical.LogicalType;
+
+@Data
+public class ColumnSchema {
Review Comment:
Please add some comments for it
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/util/SchemaChangeUtils.java:
##########
@@ -23,25 +23,31 @@
import org.apache.inlong.sort.protocol.ddl.operations.Operation;
import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.ColumnSchema;
+import org.apache.inlong.sort.schema.TableChange;
import com.google.common.base.Preconditions;
+import com.google.common.collect.Sets;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import java.util.StringJoiner;
+import java.util.stream.Collectors;
/**
* Schema-change Utils
*/
-public final class SchemaChangeUtils {
+public class SchemaChangeUtils {
Review Comment:
I don't think it is a good idea to remove final
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeInterface.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+
+public interface SchemaChangeInterface {
Review Comment:
Please add some comments for it and maybe it is better to rename it to 'SchemaChangeAPI' or 'SchemaChangeHandle'?
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
Review Comment:
Please add some comments
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+
+public abstract class SchemaChangeHelper implements SchemaChangeInterface {
Review Comment:
Please add some comments for it.
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeHelper.class);
+ private transient Catalog catalog;
+
+ private transient SupportsNamespaces asNamespaceCatalog;
+
+ public IcebergSchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper,
+ Catalog catalog,
+ SupportsNamespaces asNamespaceCatalog) {
+ super(dynamicSchemaFormat, schemaChange, policyMap, databasePattern,
+ tablePattern, exceptionPolicy, metricData, dirtySinkHelper);
+ this.catalog = catalog;
+ this.asNamespaceCatalog = asNamespaceCatalog;
+ }
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ for (Map.Entry<SchemaChangeType, List<AlterColumn>> kv : typeMap.entrySet()) {
+ SchemaChangePolicy policy = policyMap.get(kv.getKey());
+ doSchemaChangeBase(kv.getKey(), policy, originSchema);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ switch (kv.getKey()) {
+ case ADD_COLUMN:
+ doAddColumn(kv.getValue(), TableIdentifier.of(database, table));
+ break;
+ case DROP_COLUMN:
+ doDropColumn(kv.getKey(), originSchema);
+ break;
+ case RENAME_COLUMN:
+ doRenameColumn(kv.getKey(), originSchema);
+
+ break;
+ case CHANGE_COLUMN_TYPE:
+ doChangeColumnType(kv.getKey(), originSchema);
+ break;
+ default:
+ }
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Apply alter column failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Apply alter column failed, origin schema: {}", originSchema, e);
+ handleDirtyData(data, originData, database, table, DirtyType.HANDLE_ALTER_TABLE_ERROR, e);
+ }
+ }
+ }
+ }
+ }
+
+ private void doAddColumn(List<AlterColumn> alterColumns, TableIdentifier tableId) {
+ List<TableChange> tableChanges = new ArrayList<>();
+ Table table = catalog.loadTable(tableId);
+ Transaction transaction = table.newTransaction();
+
+ alterColumns.forEach(alterColumn -> {
+ Column column = alterColumn.getNewColumn();
+ LogicalType dataType = dynamicSchemaFormat.sqlType2FlinkType(column.getJdbcType());
+ TableChange.ColumnPosition position =
+ column.getPosition().getPositionType() == PositionType.FIRST ? TableChange.ColumnPosition.first()
+ : TableChange.ColumnPosition.after(column.getName());
+ TableChange.AddColumn addColumn = new TableChange.AddColumn(new String[]{column.getName()},
+ dataType, column.isNullable(), column.getComment(), position);
+ tableChanges.add(addColumn);
+ });
+ IcebergSchemaChangeUtils.applySchemaChanges(transaction.updateSchema(), tableChanges);
+ transaction.commitTransaction();
+ }
+
+ private String doChangeColumnType(SchemaChangeType type, String originSchema) {
Review Comment:
It is recommend to change it as a default implemention in the supper class.
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeHelper.class);
+ private transient Catalog catalog;
+
+ private transient SupportsNamespaces asNamespaceCatalog;
+
+ public IcebergSchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper,
+ Catalog catalog,
+ SupportsNamespaces asNamespaceCatalog) {
+ super(dynamicSchemaFormat, schemaChange, policyMap, databasePattern,
+ tablePattern, exceptionPolicy, metricData, dirtySinkHelper);
+ this.catalog = catalog;
+ this.asNamespaceCatalog = asNamespaceCatalog;
+ }
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ for (Map.Entry<SchemaChangeType, List<AlterColumn>> kv : typeMap.entrySet()) {
+ SchemaChangePolicy policy = policyMap.get(kv.getKey());
+ doSchemaChangeBase(kv.getKey(), policy, originSchema);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ switch (kv.getKey()) {
+ case ADD_COLUMN:
+ doAddColumn(kv.getValue(), TableIdentifier.of(database, table));
+ break;
+ case DROP_COLUMN:
+ doDropColumn(kv.getKey(), originSchema);
+ break;
+ case RENAME_COLUMN:
+ doRenameColumn(kv.getKey(), originSchema);
+
+ break;
+ case CHANGE_COLUMN_TYPE:
+ doChangeColumnType(kv.getKey(), originSchema);
+ break;
+ default:
+ }
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Apply alter column failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Apply alter column failed, origin schema: {}", originSchema, e);
+ handleDirtyData(data, originData, database, table, DirtyType.HANDLE_ALTER_TABLE_ERROR, e);
+ }
+ }
+ }
+ }
+ }
+
+ private void doAddColumn(List<AlterColumn> alterColumns, TableIdentifier tableId) {
+ List<TableChange> tableChanges = new ArrayList<>();
+ Table table = catalog.loadTable(tableId);
+ Transaction transaction = table.newTransaction();
+
+ alterColumns.forEach(alterColumn -> {
+ Column column = alterColumn.getNewColumn();
+ LogicalType dataType = dynamicSchemaFormat.sqlType2FlinkType(column.getJdbcType());
+ TableChange.ColumnPosition position =
+ column.getPosition().getPositionType() == PositionType.FIRST ? TableChange.ColumnPosition.first()
+ : TableChange.ColumnPosition.after(column.getName());
+ TableChange.AddColumn addColumn = new TableChange.AddColumn(new String[]{column.getName()},
+ dataType, column.isNullable(), column.getComment(), position);
+ tableChanges.add(addColumn);
+ });
+ IcebergSchemaChangeUtils.applySchemaChanges(transaction.updateSchema(), tableChanges);
+ transaction.commitTransaction();
+ }
+
+ private String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doRenameColumn(SchemaChangeType type, String originSchema) {
Review Comment:
It is recommend to change it as a default implemention in the supper class.
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeHelper.class);
+ private transient Catalog catalog;
+
+ private transient SupportsNamespaces asNamespaceCatalog;
+
+ public IcebergSchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper,
+ Catalog catalog,
+ SupportsNamespaces asNamespaceCatalog) {
+ super(dynamicSchemaFormat, schemaChange, policyMap, databasePattern,
+ tablePattern, exceptionPolicy, metricData, dirtySinkHelper);
+ this.catalog = catalog;
+ this.asNamespaceCatalog = asNamespaceCatalog;
+ }
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ for (Map.Entry<SchemaChangeType, List<AlterColumn>> kv : typeMap.entrySet()) {
+ SchemaChangePolicy policy = policyMap.get(kv.getKey());
+ doSchemaChangeBase(kv.getKey(), policy, originSchema);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ switch (kv.getKey()) {
+ case ADD_COLUMN:
+ doAddColumn(kv.getValue(), TableIdentifier.of(database, table));
+ break;
+ case DROP_COLUMN:
+ doDropColumn(kv.getKey(), originSchema);
+ break;
+ case RENAME_COLUMN:
+ doRenameColumn(kv.getKey(), originSchema);
+
+ break;
+ case CHANGE_COLUMN_TYPE:
+ doChangeColumnType(kv.getKey(), originSchema);
+ break;
+ default:
+ }
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Apply alter column failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Apply alter column failed, origin schema: {}", originSchema, e);
+ handleDirtyData(data, originData, database, table, DirtyType.HANDLE_ALTER_TABLE_ERROR, e);
+ }
+ }
+ }
+ }
+ }
+
+ private void doAddColumn(List<AlterColumn> alterColumns, TableIdentifier tableId) {
+ List<TableChange> tableChanges = new ArrayList<>();
+ Table table = catalog.loadTable(tableId);
+ Transaction transaction = table.newTransaction();
+
+ alterColumns.forEach(alterColumn -> {
+ Column column = alterColumn.getNewColumn();
+ LogicalType dataType = dynamicSchemaFormat.sqlType2FlinkType(column.getJdbcType());
+ TableChange.ColumnPosition position =
+ column.getPosition().getPositionType() == PositionType.FIRST ? TableChange.ColumnPosition.first()
+ : TableChange.ColumnPosition.after(column.getName());
+ TableChange.AddColumn addColumn = new TableChange.AddColumn(new String[]{column.getName()},
+ dataType, column.isNullable(), column.getComment(), position);
+ tableChanges.add(addColumn);
+ });
+ IcebergSchemaChangeUtils.applySchemaChanges(transaction.updateSchema(), tableChanges);
+ transaction.commitTransaction();
+ }
+
+ private String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doRenameColumn(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doDropColumn(SchemaChangeType type, String originSchema) {
Review Comment:
It is recommend to change it as a default implemention in the supper class.
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeHelper.class);
+ private transient Catalog catalog;
+
+ private transient SupportsNamespaces asNamespaceCatalog;
+
+ public IcebergSchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper,
+ Catalog catalog,
+ SupportsNamespaces asNamespaceCatalog) {
+ super(dynamicSchemaFormat, schemaChange, policyMap, databasePattern,
+ tablePattern, exceptionPolicy, metricData, dirtySinkHelper);
+ this.catalog = catalog;
+ this.asNamespaceCatalog = asNamespaceCatalog;
+ }
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ for (Map.Entry<SchemaChangeType, List<AlterColumn>> kv : typeMap.entrySet()) {
+ SchemaChangePolicy policy = policyMap.get(kv.getKey());
+ doSchemaChangeBase(kv.getKey(), policy, originSchema);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ switch (kv.getKey()) {
+ case ADD_COLUMN:
+ doAddColumn(kv.getValue(), TableIdentifier.of(database, table));
+ break;
+ case DROP_COLUMN:
+ doDropColumn(kv.getKey(), originSchema);
+ break;
+ case RENAME_COLUMN:
+ doRenameColumn(kv.getKey(), originSchema);
+
+ break;
+ case CHANGE_COLUMN_TYPE:
+ doChangeColumnType(kv.getKey(), originSchema);
+ break;
+ default:
+ }
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Apply alter column failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Apply alter column failed, origin schema: {}", originSchema, e);
+ handleDirtyData(data, originData, database, table, DirtyType.HANDLE_ALTER_TABLE_ERROR, e);
+ }
+ }
+ }
+ }
+ }
+
+ private void doAddColumn(List<AlterColumn> alterColumns, TableIdentifier tableId) {
+ List<TableChange> tableChanges = new ArrayList<>();
+ Table table = catalog.loadTable(tableId);
+ Transaction transaction = table.newTransaction();
+
+ alterColumns.forEach(alterColumn -> {
+ Column column = alterColumn.getNewColumn();
+ LogicalType dataType = dynamicSchemaFormat.sqlType2FlinkType(column.getJdbcType());
+ TableChange.ColumnPosition position =
+ column.getPosition().getPositionType() == PositionType.FIRST ? TableChange.ColumnPosition.first()
+ : TableChange.ColumnPosition.after(column.getName());
+ TableChange.AddColumn addColumn = new TableChange.AddColumn(new String[]{column.getName()},
+ dataType, column.isNullable(), column.getComment(), position);
+ tableChanges.add(addColumn);
+ });
+ IcebergSchemaChangeUtils.applySchemaChanges(transaction.updateSchema(), tableChanges);
+ transaction.commitTransaction();
+ }
+
+ private String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doRenameColumn(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doDropColumn(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private void doSchemaChangeBase(SchemaChangeType type, SchemaChangePolicy policy, String schema) {
+ if (policy == null) {
+ LOGGER.warn("Unsupported for {}: {}", type, schema);
+ return;
+ }
+ switch (policy) {
+ case LOG:
+ LOGGER.warn("Unsupported for {}: {}", type, schema);
+ break;
+ case ERROR:
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, schema));
+ default:
+ }
+ }
+
+ @Override
+ public void doCreateTable(byte[] originData, String database, String table, SchemaChangeType type,
+ String originSchema, JsonNode data, CreateTableOperation operation) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ TableIdentifier tableId = TableIdentifier.of(database, table);
+ List<String> pkListStr = dynamicSchemaFormat.extractPrimaryKeyNames(data);
+ RowType rowType = dynamicSchemaFormat.extractSchema(data, pkListStr);
+ Schema schema = FlinkSchemaUtil.convert(FlinkSchemaUtil.toSchema(rowType));
+ IcebergSchemaChangeUtils.createTable(catalog, tableId, asNamespaceCatalog, schema);
+ return;
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Drop column failed, origin schema: %s", originSchema), e);
+ }
+ handleDirtyData(data, originData, database, table, DirtyType.CREATE_TABLE_ERROR, e);
+ return;
+ }
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+
+ @Override
+ public void doDropTable(SchemaChangeType type, String originSchema) {
Review Comment:
It is recommend to change it as a default implemention in the supper class.
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeHelper.class);
+ private transient Catalog catalog;
+
+ private transient SupportsNamespaces asNamespaceCatalog;
+
+ public IcebergSchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper,
+ Catalog catalog,
+ SupportsNamespaces asNamespaceCatalog) {
+ super(dynamicSchemaFormat, schemaChange, policyMap, databasePattern,
+ tablePattern, exceptionPolicy, metricData, dirtySinkHelper);
+ this.catalog = catalog;
+ this.asNamespaceCatalog = asNamespaceCatalog;
+ }
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ for (Map.Entry<SchemaChangeType, List<AlterColumn>> kv : typeMap.entrySet()) {
+ SchemaChangePolicy policy = policyMap.get(kv.getKey());
+ doSchemaChangeBase(kv.getKey(), policy, originSchema);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ switch (kv.getKey()) {
+ case ADD_COLUMN:
+ doAddColumn(kv.getValue(), TableIdentifier.of(database, table));
+ break;
+ case DROP_COLUMN:
+ doDropColumn(kv.getKey(), originSchema);
+ break;
+ case RENAME_COLUMN:
+ doRenameColumn(kv.getKey(), originSchema);
+
+ break;
+ case CHANGE_COLUMN_TYPE:
+ doChangeColumnType(kv.getKey(), originSchema);
+ break;
+ default:
+ }
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Apply alter column failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Apply alter column failed, origin schema: {}", originSchema, e);
+ handleDirtyData(data, originData, database, table, DirtyType.HANDLE_ALTER_TABLE_ERROR, e);
+ }
+ }
+ }
+ }
+ }
+
+ private void doAddColumn(List<AlterColumn> alterColumns, TableIdentifier tableId) {
+ List<TableChange> tableChanges = new ArrayList<>();
+ Table table = catalog.loadTable(tableId);
+ Transaction transaction = table.newTransaction();
+
+ alterColumns.forEach(alterColumn -> {
+ Column column = alterColumn.getNewColumn();
+ LogicalType dataType = dynamicSchemaFormat.sqlType2FlinkType(column.getJdbcType());
+ TableChange.ColumnPosition position =
+ column.getPosition().getPositionType() == PositionType.FIRST ? TableChange.ColumnPosition.first()
+ : TableChange.ColumnPosition.after(column.getName());
+ TableChange.AddColumn addColumn = new TableChange.AddColumn(new String[]{column.getName()},
+ dataType, column.isNullable(), column.getComment(), position);
+ tableChanges.add(addColumn);
+ });
+ IcebergSchemaChangeUtils.applySchemaChanges(transaction.updateSchema(), tableChanges);
+ transaction.commitTransaction();
+ }
+
+ private String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doRenameColumn(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doDropColumn(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private void doSchemaChangeBase(SchemaChangeType type, SchemaChangePolicy policy, String schema) {
+ if (policy == null) {
+ LOGGER.warn("Unsupported for {}: {}", type, schema);
+ return;
+ }
+ switch (policy) {
+ case LOG:
+ LOGGER.warn("Unsupported for {}: {}", type, schema);
+ break;
+ case ERROR:
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, schema));
+ default:
+ }
+ }
+
+ @Override
+ public void doCreateTable(byte[] originData, String database, String table, SchemaChangeType type,
+ String originSchema, JsonNode data, CreateTableOperation operation) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ TableIdentifier tableId = TableIdentifier.of(database, table);
+ List<String> pkListStr = dynamicSchemaFormat.extractPrimaryKeyNames(data);
+ RowType rowType = dynamicSchemaFormat.extractSchema(data, pkListStr);
+ Schema schema = FlinkSchemaUtil.convert(FlinkSchemaUtil.toSchema(rowType));
+ IcebergSchemaChangeUtils.createTable(catalog, tableId, asNamespaceCatalog, schema);
+ return;
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Drop column failed, origin schema: %s", originSchema), e);
+ }
+ handleDirtyData(data, originData, database, table, DirtyType.CREATE_TABLE_ERROR, e);
+ return;
+ }
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+
+ @Override
+ public void doDropTable(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return;
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+
+ @Override
+ public void doRenameTable(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.RENAME_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return;
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+
+ @Override
+ public void doTruncateTable(SchemaChangeType type, String originSchema) {
Review Comment:
It is recommend to change it as a default implemention in the supper class.
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeHelper.class);
+ private transient Catalog catalog;
+
+ private transient SupportsNamespaces asNamespaceCatalog;
+
+ public IcebergSchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper,
+ Catalog catalog,
+ SupportsNamespaces asNamespaceCatalog) {
+ super(dynamicSchemaFormat, schemaChange, policyMap, databasePattern,
+ tablePattern, exceptionPolicy, metricData, dirtySinkHelper);
+ this.catalog = catalog;
+ this.asNamespaceCatalog = asNamespaceCatalog;
+ }
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ for (Map.Entry<SchemaChangeType, List<AlterColumn>> kv : typeMap.entrySet()) {
+ SchemaChangePolicy policy = policyMap.get(kv.getKey());
+ doSchemaChangeBase(kv.getKey(), policy, originSchema);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ switch (kv.getKey()) {
+ case ADD_COLUMN:
+ doAddColumn(kv.getValue(), TableIdentifier.of(database, table));
+ break;
+ case DROP_COLUMN:
+ doDropColumn(kv.getKey(), originSchema);
+ break;
+ case RENAME_COLUMN:
+ doRenameColumn(kv.getKey(), originSchema);
+
+ break;
+ case CHANGE_COLUMN_TYPE:
+ doChangeColumnType(kv.getKey(), originSchema);
+ break;
+ default:
+ }
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Apply alter column failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Apply alter column failed, origin schema: {}", originSchema, e);
+ handleDirtyData(data, originData, database, table, DirtyType.HANDLE_ALTER_TABLE_ERROR, e);
+ }
+ }
+ }
+ }
+ }
+
+ private void doAddColumn(List<AlterColumn> alterColumns, TableIdentifier tableId) {
+ List<TableChange> tableChanges = new ArrayList<>();
+ Table table = catalog.loadTable(tableId);
+ Transaction transaction = table.newTransaction();
+
+ alterColumns.forEach(alterColumn -> {
+ Column column = alterColumn.getNewColumn();
+ LogicalType dataType = dynamicSchemaFormat.sqlType2FlinkType(column.getJdbcType());
+ TableChange.ColumnPosition position =
+ column.getPosition().getPositionType() == PositionType.FIRST ? TableChange.ColumnPosition.first()
+ : TableChange.ColumnPosition.after(column.getName());
+ TableChange.AddColumn addColumn = new TableChange.AddColumn(new String[]{column.getName()},
+ dataType, column.isNullable(), column.getComment(), position);
+ tableChanges.add(addColumn);
+ });
+ IcebergSchemaChangeUtils.applySchemaChanges(transaction.updateSchema(), tableChanges);
+ transaction.commitTransaction();
+ }
+
+ private String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doRenameColumn(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doDropColumn(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private void doSchemaChangeBase(SchemaChangeType type, SchemaChangePolicy policy, String schema) {
Review Comment:
It is recommend to change it as a default implemention in the supper class.
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeHelper.class);
+ private transient Catalog catalog;
+
+ private transient SupportsNamespaces asNamespaceCatalog;
+
+ public IcebergSchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper,
+ Catalog catalog,
+ SupportsNamespaces asNamespaceCatalog) {
+ super(dynamicSchemaFormat, schemaChange, policyMap, databasePattern,
+ tablePattern, exceptionPolicy, metricData, dirtySinkHelper);
+ this.catalog = catalog;
+ this.asNamespaceCatalog = asNamespaceCatalog;
+ }
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ for (Map.Entry<SchemaChangeType, List<AlterColumn>> kv : typeMap.entrySet()) {
+ SchemaChangePolicy policy = policyMap.get(kv.getKey());
+ doSchemaChangeBase(kv.getKey(), policy, originSchema);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ switch (kv.getKey()) {
+ case ADD_COLUMN:
+ doAddColumn(kv.getValue(), TableIdentifier.of(database, table));
+ break;
+ case DROP_COLUMN:
+ doDropColumn(kv.getKey(), originSchema);
+ break;
+ case RENAME_COLUMN:
+ doRenameColumn(kv.getKey(), originSchema);
+
+ break;
+ case CHANGE_COLUMN_TYPE:
+ doChangeColumnType(kv.getKey(), originSchema);
+ break;
+ default:
+ }
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Apply alter column failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Apply alter column failed, origin schema: {}", originSchema, e);
+ handleDirtyData(data, originData, database, table, DirtyType.HANDLE_ALTER_TABLE_ERROR, e);
+ }
+ }
+ }
+ }
+ }
+
+ private void doAddColumn(List<AlterColumn> alterColumns, TableIdentifier tableId) {
+ List<TableChange> tableChanges = new ArrayList<>();
+ Table table = catalog.loadTable(tableId);
+ Transaction transaction = table.newTransaction();
+
+ alterColumns.forEach(alterColumn -> {
+ Column column = alterColumn.getNewColumn();
+ LogicalType dataType = dynamicSchemaFormat.sqlType2FlinkType(column.getJdbcType());
+ TableChange.ColumnPosition position =
+ column.getPosition().getPositionType() == PositionType.FIRST ? TableChange.ColumnPosition.first()
+ : TableChange.ColumnPosition.after(column.getName());
+ TableChange.AddColumn addColumn = new TableChange.AddColumn(new String[]{column.getName()},
+ dataType, column.isNullable(), column.getComment(), position);
+ tableChanges.add(addColumn);
+ });
+ IcebergSchemaChangeUtils.applySchemaChanges(transaction.updateSchema(), tableChanges);
+ transaction.commitTransaction();
+ }
+
+ private String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doRenameColumn(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private String doDropColumn(SchemaChangeType type, String originSchema) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return null;
+ }
+
+ private void doSchemaChangeBase(SchemaChangeType type, SchemaChangePolicy policy, String schema) {
+ if (policy == null) {
+ LOGGER.warn("Unsupported for {}: {}", type, schema);
+ return;
+ }
+ switch (policy) {
+ case LOG:
+ LOGGER.warn("Unsupported for {}: {}", type, schema);
+ break;
+ case ERROR:
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, schema));
+ default:
+ }
+ }
+
+ @Override
+ public void doCreateTable(byte[] originData, String database, String table, SchemaChangeType type,
+ String originSchema, JsonNode data, CreateTableOperation operation) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ TableIdentifier tableId = TableIdentifier.of(database, table);
+ List<String> pkListStr = dynamicSchemaFormat.extractPrimaryKeyNames(data);
+ RowType rowType = dynamicSchemaFormat.extractSchema(data, pkListStr);
+ Schema schema = FlinkSchemaUtil.convert(FlinkSchemaUtil.toSchema(rowType));
+ IcebergSchemaChangeUtils.createTable(catalog, tableId, asNamespaceCatalog, schema);
+ return;
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Drop column failed, origin schema: %s", originSchema), e);
+ }
+ handleDirtyData(data, originData, database, table, DirtyType.CREATE_TABLE_ERROR, e);
+ return;
+ }
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+
+ @Override
+ public void doDropTable(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ return;
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+
+ @Override
+ public void doRenameTable(SchemaChangeType type, String originSchema) {
Review Comment:
It is recommend to change it as a default implemention in the supper class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1217613284
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/pom.xml:
##########
@@ -143,8 +143,8 @@
<shadedPattern>org.apache.inlong.sort.iceberg.shaded.com.amazonaws</shadedPattern>
</relocation>
<relocation>
- <pattern>org.apache.inlong.sort.base</pattern>
- <shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort.base</shadedPattern>
+ <pattern>org.apache.inlong.sort</pattern>
+ <shadedPattern>org.apache.inlong.sort.iceberg.shaded.org.apache.inlong.sort</shadedPattern>
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214174054
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Schema change helper
+ * */
+public abstract class SchemaChangeHelper implements SchemaChangeHandle {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SchemaChangeHelper.class);
+ private final boolean schemaChange;
+ protected final Map<SchemaChangeType, SchemaChangePolicy> policyMap;
+ protected final JsonDynamicSchemaFormat dynamicSchemaFormat;
+ private final String databasePattern;
+ private final String tablePattern;
+ protected final SchemaUpdateExceptionPolicy exceptionPolicy;
+ private final SinkTableMetricData metricData;
+ private final DirtySinkHelper<Object> dirtySinkHelper;
+
+ public SchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper) {
+ this.dynamicSchemaFormat = Preconditions.checkNotNull(dynamicSchemaFormat, "dynamicSchemaFormat is null");
+ this.schemaChange = schemaChange;
+ this.policyMap = policyMap;
+ this.databasePattern = databasePattern;
+ this.tablePattern = tablePattern;
+ this.exceptionPolicy = exceptionPolicy;
+ this.metricData = metricData;
+ this.dirtySinkHelper = dirtySinkHelper;
+ }
+
+ public void process(byte[] originData, JsonNode data) {
+ // 1. Extract the schema change type from the data;
+ if (!schemaChange) {
+ return;
+ }
+ String database;
+ String table;
+ try {
+ database = dynamicSchemaFormat.parse(data, databasePattern);
+ table = dynamicSchemaFormat.parse(data, tablePattern);
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Parse database, table from origin data failed, origin data: %s",
+ new String(originData)),
+ e);
+ }
+ LOGGER.warn("Parse database, table from origin data failed, origin data: {}", new String(originData), e);
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.LOG_WITH_IGNORE) {
+ dirtySinkHelper.invoke(new String(originData), DirtyType.JSON_PROCESS_ERROR, e);
+ }
+ if (metricData != null) {
+ metricData.invokeDirty(1, originData.length);
+ }
+ return;
+ }
+ Operation operation;
+ try {
+ JsonNode operationNode = Preconditions.checkNotNull(data.get("operation"),
+ "Operation node is null");
+ operation = Preconditions.checkNotNull(
+ dynamicSchemaFormat.objectMapper.convertValue(operationNode, new TypeReference<Operation>() {
+ }), "Operation is null");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract Operation from origin data failed,origin data: %s", data), e);
+ }
+ LOGGER.warn("Extract Operation from origin data failed,origin data: {}", data, e);
+ handleDirtyData(data, originData, database, table, DirtyType.JSON_PROCESS_ERROR, e);
+ return;
+ }
+ String originSchema = dynamicSchemaFormat.extractDDL(data);
+ SchemaChangeType type = SchemaChangeUtils.extractSchemaChangeType(operation);
+ if (type == null) {
+ LOGGER.warn("Unsupported for schema-change: {}", originSchema);
+ return;
+ }
+
+ // 2. Apply schema change;
+ switch (type) {
+ case ALTER:
+ handleAlterOperation(database, table, originData, originSchema, data, (AlterOperation) operation);
+ break;
+ case CREATE_TABLE:
+ doCreateTable(originData, database, table, type, originSchema, data, (CreateTableOperation) operation);
+ break;
+ case DROP_TABLE:
+ doDropTable(type, originSchema);
+ break;
+ case RENAME_TABLE:
+ doRenameTable(type, originSchema);
+ break;
+ case TRUNCATE_TABLE:
+ doTruncateTable(type, originSchema);
+ break;
+ default:
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ }
+ }
+
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ doAlterOperation(database, table, originData, originSchema, data, typeMap);
+ }
+ }
+
+ @Override
+ public String doAddColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
Review Comment:
Doris connector may use the return value of this method
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHelper.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.Operation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Schema change helper
+ * */
+public abstract class SchemaChangeHelper implements SchemaChangeHandle {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SchemaChangeHelper.class);
+ private final boolean schemaChange;
+ protected final Map<SchemaChangeType, SchemaChangePolicy> policyMap;
+ protected final JsonDynamicSchemaFormat dynamicSchemaFormat;
+ private final String databasePattern;
+ private final String tablePattern;
+ protected final SchemaUpdateExceptionPolicy exceptionPolicy;
+ private final SinkTableMetricData metricData;
+ private final DirtySinkHelper<Object> dirtySinkHelper;
+
+ public SchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper) {
+ this.dynamicSchemaFormat = Preconditions.checkNotNull(dynamicSchemaFormat, "dynamicSchemaFormat is null");
+ this.schemaChange = schemaChange;
+ this.policyMap = policyMap;
+ this.databasePattern = databasePattern;
+ this.tablePattern = tablePattern;
+ this.exceptionPolicy = exceptionPolicy;
+ this.metricData = metricData;
+ this.dirtySinkHelper = dirtySinkHelper;
+ }
+
+ public void process(byte[] originData, JsonNode data) {
+ // 1. Extract the schema change type from the data;
+ if (!schemaChange) {
+ return;
+ }
+ String database;
+ String table;
+ try {
+ database = dynamicSchemaFormat.parse(data, databasePattern);
+ table = dynamicSchemaFormat.parse(data, tablePattern);
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Parse database, table from origin data failed, origin data: %s",
+ new String(originData)),
+ e);
+ }
+ LOGGER.warn("Parse database, table from origin data failed, origin data: {}", new String(originData), e);
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.LOG_WITH_IGNORE) {
+ dirtySinkHelper.invoke(new String(originData), DirtyType.JSON_PROCESS_ERROR, e);
+ }
+ if (metricData != null) {
+ metricData.invokeDirty(1, originData.length);
+ }
+ return;
+ }
+ Operation operation;
+ try {
+ JsonNode operationNode = Preconditions.checkNotNull(data.get("operation"),
+ "Operation node is null");
+ operation = Preconditions.checkNotNull(
+ dynamicSchemaFormat.objectMapper.convertValue(operationNode, new TypeReference<Operation>() {
+ }), "Operation is null");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract Operation from origin data failed,origin data: %s", data), e);
+ }
+ LOGGER.warn("Extract Operation from origin data failed,origin data: {}", data, e);
+ handleDirtyData(data, originData, database, table, DirtyType.JSON_PROCESS_ERROR, e);
+ return;
+ }
+ String originSchema = dynamicSchemaFormat.extractDDL(data);
+ SchemaChangeType type = SchemaChangeUtils.extractSchemaChangeType(operation);
+ if (type == null) {
+ LOGGER.warn("Unsupported for schema-change: {}", originSchema);
+ return;
+ }
+
+ // 2. Apply schema change;
+ switch (type) {
+ case ALTER:
+ handleAlterOperation(database, table, originData, originSchema, data, (AlterOperation) operation);
+ break;
+ case CREATE_TABLE:
+ doCreateTable(originData, database, table, type, originSchema, data, (CreateTableOperation) operation);
+ break;
+ case DROP_TABLE:
+ doDropTable(type, originSchema);
+ break;
+ case RENAME_TABLE:
+ doRenameTable(type, originSchema);
+ break;
+ case TRUNCATE_TABLE:
+ doTruncateTable(type, originSchema);
+ break;
+ default:
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ }
+ }
+
+ @Override
+ public void handleAlterOperation(String database, String table, byte[] originData,
+ String originSchema, JsonNode data, AlterOperation operation) {
+ if (operation.getAlterColumns() == null || operation.getAlterColumns().isEmpty()) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Alter columns is empty, origin schema: %s", originSchema));
+ }
+ LOGGER.warn("Alter columns is empty, origin schema: {}", originSchema);
+ return;
+ }
+
+ Map<SchemaChangeType, List<AlterColumn>> typeMap = new LinkedHashMap<>();
+ for (AlterColumn alterColumn : operation.getAlterColumns()) {
+ Set<SchemaChangeType> types = null;
+ try {
+ types = SchemaChangeUtils.extractSchemaChangeType(alterColumn);
+ Preconditions.checkState(!types.isEmpty(), "Schema change types is empty");
+ } catch (Exception e) {
+ if (exceptionPolicy == SchemaUpdateExceptionPolicy.THROW_WITH_STOP) {
+ throw new SchemaChangeHandleException(
+ String.format("Extract schema change type failed, origin schema: %s", originSchema), e);
+ }
+ LOGGER.warn("Extract schema change type failed, origin schema: {}", originSchema, e);
+ }
+ if (types == null) {
+ continue;
+ }
+ if (types.size() == 1) {
+ SchemaChangeType type = types.stream().findFirst().get();
+ typeMap.computeIfAbsent(type, k -> new ArrayList<>()).add(alterColumn);
+ } else {
+ // Handle change column, it only exists change column type and rename column in this scenario for now.
+ for (SchemaChangeType type : types) {
+ SchemaChangePolicy policy = policyMap.get(type);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ LOGGER.warn("Unsupported for {}: {}", type, originSchema);
+ } else {
+ doSchemaChangeBase(type, policy, originSchema);
+ }
+ }
+ }
+ }
+
+ if (!typeMap.isEmpty()) {
+ doAlterOperation(database, table, originData, originSchema, data, typeMap);
+ }
+ }
+
+ @Override
+ public String doAddColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doChangeColumnType(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doRenameColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
+
+ @Override
+ public String doDropColumn(SchemaChangeType type, String originSchema) {
+ SchemaChangePolicy policy = policyMap.get(SchemaChangeType.DROP_TABLE);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ throw new SchemaChangeHandleException(String.format("Unsupported for %s: %s", type, originSchema));
+ }
+ doSchemaChangeBase(type, policy, originSchema);
+ return null;
+ }
Review Comment:
Doris connector may use the return value of this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214099394
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHandle.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+
+import java.util.List;
+import java.util.Map;
+
+public interface SchemaChangeHandle {
+
+ void process(byte[] originData, JsonNode data);
+
+ void handleAlterOperation(String database, String table, byte[] originData, String originSchema,
+ JsonNode data, AlterOperation operation);
+
+ void doAlterOperation(String database, String table, byte[] originData, String originSchema, JsonNode data,
+ Map<SchemaChangeType, List<AlterColumn>> typeMap);
Review Comment:
I guess Map iterates without order
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214126192
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/schema/IcebergSchemaChangeHelper.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.schema;
+
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.schema.SchemaChangeHandleException;
+import org.apache.inlong.sort.base.schema.SchemaChangeHelper;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.iceberg.sink.multiple.IcebergSchemaChangeUtils;
+import org.apache.inlong.sort.protocol.ddl.enums.PositionType;
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.expressions.Column;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangePolicy;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+import org.apache.inlong.sort.schema.TableChange;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.Transaction;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Iceberg schema change helper
+ * */
+public class IcebergSchemaChangeHelper extends SchemaChangeHelper {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeHelper.class);
+ private transient Catalog catalog;
+
+ private transient SupportsNamespaces asNamespaceCatalog;
+
+ public IcebergSchemaChangeHelper(JsonDynamicSchemaFormat dynamicSchemaFormat, boolean schemaChange,
+ Map<SchemaChangeType, SchemaChangePolicy> policyMap, String databasePattern, String tablePattern,
+ SchemaUpdateExceptionPolicy exceptionPolicy,
+ SinkTableMetricData metricData, DirtySinkHelper<Object> dirtySinkHelper,
+ Catalog catalog,
+ SupportsNamespaces asNamespaceCatalog) {
+ super(dynamicSchemaFormat, schemaChange, policyMap, databasePattern,
+ tablePattern, exceptionPolicy, metricData, dirtySinkHelper);
+ this.catalog = catalog;
+ this.asNamespaceCatalog = asNamespaceCatalog;
+ }
+ @Override
+ public void doAlterOperation(String database, String table, byte[] originData, String originSchema, JsonNode data,
+ Map<SchemaChangeType, List<AlterColumn>> typeMap) {
+ for (Map.Entry<SchemaChangeType, List<AlterColumn>> kv : typeMap.entrySet()) {
+ SchemaChangePolicy policy = policyMap.get(kv.getKey());
+ doSchemaChangeBase(kv.getKey(), policy, originSchema);
+ if (policy == SchemaChangePolicy.ENABLE) {
+ try {
+ switch (kv.getKey()) {
+ case ADD_COLUMN:
+ doAddColumn(kv.getValue(), TableIdentifier.of(database, table));
+ break;
+ case DROP_COLUMN:
+ doDropColumn(kv.getKey(), originSchema);
+ break;
+ case RENAME_COLUMN:
+ doRenameColumn(kv.getKey(), originSchema);
+ break;
+ case CHANGE_COLUMN_TYPE:
+ doChangeColumnType(kv.getKey(), originSchema);
Review Comment:
doSchemaChangeBase is called before and also inside doChangeColumnType, why is that and any way to improve this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang merged PR #8141:
URL: https://github.com/apache/inlong/pull/8141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214172097
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/IcebergSchemaChangeUtils.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.iceberg.sink.multiple;
+
+import org.apache.inlong.sort.iceberg.FlinkTypeToType;
+import org.apache.inlong.sort.schema.TableChange;
+import org.apache.inlong.sort.util.SchemaChangeUtils;
+
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.UpdateSchema;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.SupportsNamespaces;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.AlreadyExistsException;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+
+public class IcebergSchemaChangeUtils extends SchemaChangeUtils {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IcebergSchemaChangeUtils.class);
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ public static void createTable(Catalog catalog, TableIdentifier tableId, SupportsNamespaces asNamespaceCatalog,
+ Schema schema) {
+ if (!catalog.tableExists(tableId)) {
Review Comment:
It is not good to judge whether it exists every time it is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1214173640
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeHandle.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.protocol.ddl.expressions.AlterColumn;
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+
+import java.util.List;
+import java.util.Map;
+
+public interface SchemaChangeHandle {
+
+ void process(byte[] originData, JsonNode data);
+
+ void handleAlterOperation(String database, String table, byte[] originData, String originSchema,
+ JsonNode data, AlterOperation operation);
+
+ void doAlterOperation(String database, String table, byte[] originData, String originSchema, JsonNode data,
+ Map<SchemaChangeType, List<AlterColumn>> typeMap);
+ String doAddColumn(SchemaChangeType type, String originSchema);
+
+ String doChangeColumnType(SchemaChangeType type, String originSchema);
+
+ String doRenameColumn(SchemaChangeType type, String originSchema);
Review Comment:
doris connector may use the return value of this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1217403544
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/iceberg/pom.xml:
##########
@@ -88,6 +88,12 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.inlong</groupId>
+ <artifactId>sort-common</artifactId>
+ <version>1.8.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #8141: [INLONG-8140][Sort] Support data inference schema change type and iceberg ddl change
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #8141:
URL: https://github.com/apache/inlong/pull/8141#discussion_r1213936734
##########
inlong-sort/sort-flink/base/src/main/java/org/apache/inlong/sort/base/schema/SchemaChangeInterface.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base.schema;
+
+import org.apache.inlong.sort.protocol.ddl.operations.AlterOperation;
+import org.apache.inlong.sort.protocol.ddl.operations.CreateTableOperation;
+import org.apache.inlong.sort.protocol.enums.SchemaChangeType;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+
+public interface SchemaChangeInterface {
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org