You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "qidian99 (via GitHub)" <gi...@apache.org> on 2023/05/23 13:13:42 UTC
[GitHub] [incubator-paimon] qidian99 opened a new pull request, #1212: [feature] Support multi-table write operator
qidian99 opened a new pull request, #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- What is the purpose of the change, or the associated issue -->
This PR implement a multiplexed Paimon sink writer and committer to support multi-table write as mentioned in #1181.
### Tests
<!-- List UT and IT cases to verify this change -->
CdcRecordStoreMultiWriteOperatorTest
### API and Format
<!-- Does this change affect API or storage format -->
Yes.
### Documentation
<!-- Does this change introduce a new feature -->
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203293916
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/MultiTableCommittable.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.catalog.Identifier;
+
+/**
+ * MultiTableCommittable produced by {@link PrepareCommitOperator}. This type of Committable will
+ * only be produced by multiplexed operators that handles data from multiple tables. Thus, the
+ * database, table, and commit user of each table is included in the committable.
+ */
+public class MultiTableCommittable extends Committable {
+
+ private final String database;
+ private final String table;
+ private final String commitUser;
+
+ public MultiTableCommittable(
+ String database,
+ String table,
+ String commitUser,
Review Comment:
Yes, it's possible. There are several places where commit user is used.
1. When the write operator is assembly into job graph, a random uid is set to be the commit user id.
2. When the job is deployed and running, it will try to recover the previous commit user from Flink state to make the commit user consistent accross different Flink job attempts.
3. When the CommitterOperator do commit, it will use create the committer with commit user.
So in this case we user the same commit user for all tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203337804
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecordStoreMultiWriteOperator.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.flink.sink.Committable;
+import org.apache.paimon.flink.sink.MultiTableCommittable;
+import org.apache.paimon.flink.sink.PrepareCommitOperator;
+import org.apache.paimon.flink.sink.StateUtils;
+import org.apache.paimon.flink.sink.StoreSinkWrite;
+import org.apache.paimon.flink.sink.StoreSinkWriteState;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static org.apache.paimon.flink.sink.cdc.CdcRecordStoreWriteOperator.RETRY_SLEEP_TIME;
+
+/**
+ * A {@link PrepareCommitOperator} to write {@link CdcRecord}. Record schema may change. If current
+ * known schema does not fit record schema, this operator will wait for schema changes.
+ */
+public class CdcRecordStoreMultiWriteOperator extends PrepareCommitOperator<MultiplexCdcRecord> {
+
+ private static final long serialVersionUID = 1L;
+ private final Catalog catalog;
+ private final StoreSinkWrite.Provider storeSinkWriteProvider;
+ private final String initialCommitUser;
+ private FileStoreTable table;
+ private Map<Identifier, FileStoreTable> tables;
+ private StoreSinkWriteState state;
+ private StoreSinkWrite write;
+ private Map<Identifier, StoreSinkWrite> writes;
+ private String commitUser;
+
+ public CdcRecordStoreMultiWriteOperator(
+ Catalog catalog,
+ StoreSinkWrite.Provider storeSinkWriteProvider,
+ String initialCommitUser) {
+ this.catalog = catalog;
+ this.storeSinkWriteProvider = storeSinkWriteProvider;
+ this.initialCommitUser = initialCommitUser;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Each job can only have one user name and this name must be consistent across restarts.
+ // We cannot use job id as commit user name here because user may change job id by creating
+ // a savepoint, stop the job and then resume from savepoint.
+ commitUser =
+ StateUtils.getSingleValueFromState(
+ context, "commit_user_state", String.class, initialCommitUser);
+
+ state = new StoreSinkWriteState(context, (tableName, partition, bucket) -> true);
+ this.tables = new HashMap<>();
+ this.writes = new HashMap<>();
+ }
+
+ @Override
+ public void processElement(StreamRecord<MultiplexCdcRecord> element) throws Exception {
+ MultiplexCdcRecord record = element.getValue();
+
+ String databaseName = record.getDatabaseName();
+ String tableName = record.getTableName();
+ Identifier key = Identifier.create(databaseName, tableName);
+ table =
+ tables.computeIfAbsent(
+ key,
+ id -> {
+ try {
+ return (FileStoreTable) catalog.getTable(id);
+ } catch (Catalog.TableNotExistException e) {
+ return null;
+ }
+ });
+
+ if (table == null) {
Review Comment:
The process function is in a separate task, it is not a blocking manner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203199143
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/MultiTableCommittable.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.catalog.Identifier;
+
+/**
+ * MultiTableCommittable produced by {@link PrepareCommitOperator}. This type of Committable will
+ * only be produced by multiplexed operators that handles data from multiple tables. Thus, the
+ * database, table, and commit user of each table is included in the committable.
+ */
+public class MultiTableCommittable extends Committable {
Review Comment:
The extending class is necessary here becase in the base class `PrepareCommitOperator`, we have the following signature:
```java
protected abstract List<Committable> prepareCommit(boolean doCompaction, long checkpointId)
throws IOException;
```
Therefore, the new operator should also produce a subclass of committable and use another custom serializer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203506378
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/CommittableType.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+/** CommittableType produced by {@link PrepareCommitOperator}. */
+public interface CommittableType {
Review Comment:
Do we need to introduce this class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1204154971
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/CommittableStateManager.java:
##########
@@ -29,10 +27,10 @@
* Helper interface for {@link CommitterOperator}. This interface manages operator states about
* {@link org.apache.paimon.manifest.ManifestCommittable}.
*/
-public interface CommittableStateManager extends Serializable {
+public interface CommittableStateManager<GlobalCommitT> extends Serializable {
void initializeState(StateInitializationContext context, Committer committer) throws Exception;
Review Comment:
`Committer<?, GlobalCommitT>`?
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/RestoreAndFailCommittableStateManager.java:
##########
@@ -41,7 +41,8 @@
* <p>Useful for committing snapshots containing records. For example snapshots produced by table
* store writers.
*/
-public class RestoreAndFailCommittableStateManager implements CommittableStateManager {
+public class RestoreAndFailCommittableStateManager
+ implements CommittableStateManager<ManifestCommittable> {
private static final long serialVersionUID = 1L;
Review Comment:
Committer > Committer<....> all in this class.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/NoopCommittableStateManager.java:
##########
@@ -31,7 +31,7 @@
* <p>Useful for committing optional snapshots. For example COMPACT snapshots produced by a separate
* compact job.
*/
-public class NoopCommittableStateManager implements CommittableStateManager {
+public class NoopCommittableStateManager implements CommittableStateManager<ManifestCommittable> {
@Override
public void initializeState(StateInitializationContext context, Committer committer)
Review Comment:
Committer > Committer<....>
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSink.java:
##########
@@ -142,7 +145,7 @@ public DataStreamSink<?> sinkFrom(
written.transform(
GLOBAL_COMMITTER_NAME + " -> " + table.name(),
typeInfo,
- new CommitterOperator(
+ new CommitterOperator<CommitT, GlobalCommitT>(
Review Comment:
<> is ok
##########
paimon-core/src/main/java/org/apache/paimon/manifest/ManifestCommittable.java:
##########
@@ -35,6 +35,8 @@ public class ManifestCommittable {
@Nullable private final Long watermark;
private final Map<Integer, Long> logOffsets;
private final List<CommitMessage> commitMessages;
+ @Nullable private String database;
Review Comment:
revert this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203510893
##########
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/sink/cdc/CdcRecordStoreMultiWriteOperatorTest.java:
##########
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.flink.sink.MultiTableCommittable;
+import org.apache.paimon.flink.sink.MultiTableCommittableTypeInfo;
+import org.apache.paimon.flink.sink.StoreSinkWriteImpl;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.schema.Schema;
+import org.apache.paimon.schema.SchemaChange;
+import org.apache.paimon.schema.SchemaManager;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypes;
+import org.apache.paimon.types.RowKind;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.TraceableFileIO;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.JavaSerializer;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Predicate;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link CdcRecordStoreMultiWriteOperator}. */
+public class CdcRecordStoreMultiWriteOperatorTest {
Review Comment:
Can you add tests for `initializeState`? Just like `CommitterOperatorTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi merged pull request #1212: [feature] Support multi-table write operator
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203204316
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecordStoreMultiWriteOperator.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.flink.sink.Committable;
+import org.apache.paimon.flink.sink.MultiTableCommittable;
+import org.apache.paimon.flink.sink.PrepareCommitOperator;
+import org.apache.paimon.flink.sink.StateUtils;
+import org.apache.paimon.flink.sink.StoreSinkWrite;
+import org.apache.paimon.flink.sink.StoreSinkWriteState;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static org.apache.paimon.flink.sink.cdc.CdcRecordStoreWriteOperator.RETRY_SLEEP_TIME;
+
+/**
+ * A {@link PrepareCommitOperator} to write {@link CdcRecord}. Record schema may change. If current
+ * known schema does not fit record schema, this operator will wait for schema changes.
+ */
+public class CdcRecordStoreMultiWriteOperator extends PrepareCommitOperator<MultiplexCdcRecord> {
+
+ private static final long serialVersionUID = 1L;
+ private final Catalog catalog;
+ private final StoreSinkWrite.Provider storeSinkWriteProvider;
+ private final String initialCommitUser;
+ private FileStoreTable table;
+ private Map<Identifier, FileStoreTable> tables;
+ private StoreSinkWriteState state;
+ private StoreSinkWrite write;
+ private Map<Identifier, StoreSinkWrite> writes;
+ private String commitUser;
+
+ public CdcRecordStoreMultiWriteOperator(
+ Catalog catalog,
+ StoreSinkWrite.Provider storeSinkWriteProvider,
+ String initialCommitUser) {
+ this.catalog = catalog;
+ this.storeSinkWriteProvider = storeSinkWriteProvider;
+ this.initialCommitUser = initialCommitUser;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Each job can only have one user name and this name must be consistent across restarts.
+ // We cannot use job id as commit user name here because user may change job id by creating
+ // a savepoint, stop the job and then resume from savepoint.
+ commitUser =
+ StateUtils.getSingleValueFromState(
+ context, "commit_user_state", String.class, initialCommitUser);
+
+ state = new StoreSinkWriteState(context, (tableName, partition, bucket) -> true);
+ this.tables = new HashMap<>();
+ this.writes = new HashMap<>();
+ }
+
+ @Override
+ public void processElement(StreamRecord<MultiplexCdcRecord> element) throws Exception {
+ MultiplexCdcRecord record = element.getValue();
+
+ String databaseName = record.getDatabaseName();
+ String tableName = record.getTableName();
+ Identifier key = Identifier.create(databaseName, tableName);
+ table =
+ tables.computeIfAbsent(
+ key,
+ id -> {
+ try {
+ return (FileStoreTable) catalog.getTable(id);
+ } catch (Catalog.TableNotExistException e) {
+ return null;
+ }
+ });
+
+ if (table == null) {
Review Comment:
For MySQL CDC:
1. In snapshot phase, any DDL changes (including new table added) will result in Source table schema inconsistency and the job will fail, so we only consider ddl in incremental phase.
2. In incremental phase, it will only use 1 parallelism (source reader) to subscribe binlog, so the process function that create the new table in a blocking manner strictly `happen-before` this code segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] hk-lrzy commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "hk-lrzy (via GitHub)" <gi...@apache.org>.
hk-lrzy commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1230605356
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecordStoreMultiWriteOperator.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.flink.sink.MultiTableCommittable;
+import org.apache.paimon.flink.sink.PrepareCommitOperator;
+import org.apache.paimon.flink.sink.StateUtils;
+import org.apache.paimon.flink.sink.StoreSinkWrite;
+import org.apache.paimon.flink.sink.StoreSinkWriteState;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static org.apache.paimon.flink.sink.cdc.CdcRecordStoreWriteOperator.RETRY_SLEEP_TIME;
+
+/**
+ * A {@link PrepareCommitOperator} to write {@link CdcRecord}. Record schema may change. If current
+ * known schema does not fit record schema, this operator will wait for schema changes.
+ */
+public class CdcRecordStoreMultiWriteOperator
+ extends PrepareCommitOperator<CdcMultiplexRecord, MultiTableCommittable> {
+
+ private static final long serialVersionUID = 1L;
+ private final long retrySleepMillis;
+ private final StoreSinkWrite.Provider storeSinkWriteProvider;
+ private final String initialCommitUser;
+ private final Catalog.Loader catalogLoader;
+ protected Catalog catalog;
+ protected FileStoreTable table;
+ protected Map<Identifier, FileStoreTable> tables;
+ protected StoreSinkWriteState state;
+ protected StoreSinkWrite write;
+ protected Map<Identifier, StoreSinkWrite> writes;
+ protected String commitUser;
+
+ public CdcRecordStoreMultiWriteOperator(
+ Catalog.Loader catalogLoader,
+ StoreSinkWrite.Provider storeSinkWriteProvider,
+ String initialCommitUser) {
+ this.catalogLoader = catalogLoader;
+ this.storeSinkWriteProvider = storeSinkWriteProvider;
+ this.initialCommitUser = initialCommitUser;
+ this.retrySleepMillis = RETRY_SLEEP_TIME.defaultValue().toMillis();
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ catalog = catalogLoader.load();
+
+ // Each job can only have one user name and this name must be consistent across restarts.
+ // We cannot use job id as commit user name here because user may change job id by creating
+ // a savepoint, stop the job and then resume from savepoint.
+ commitUser =
+ StateUtils.getSingleValueFromState(
+ context, "commit_user_state", String.class, initialCommitUser);
+
+ state = new StoreSinkWriteState(context, (tableName, partition, bucket) -> true);
Review Comment:
Now, State will be expand when we try to scale out right? every task will store all table's states maybe .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1204917846
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecordStoreMultiWriteOperator.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.flink.sink.MultiTableCommittable;
+import org.apache.paimon.flink.sink.PrepareCommitOperator;
+import org.apache.paimon.flink.sink.StateUtils;
+import org.apache.paimon.flink.sink.StoreSinkWrite;
+import org.apache.paimon.flink.sink.StoreSinkWriteState;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static org.apache.paimon.flink.sink.cdc.CdcRecordStoreWriteOperator.RETRY_SLEEP_TIME;
+
+/**
+ * A {@link PrepareCommitOperator} to write {@link CdcRecord}. Record schema may change. If current
+ * known schema does not fit record schema, this operator will wait for schema changes.
+ */
+public class CdcRecordStoreMultiWriteOperator
+ extends PrepareCommitOperator<CdcMultiplexRecord, MultiTableCommittable> {
+
+ private static final long serialVersionUID = 1L;
+ private final long retrySleepMillis;
+ private final StoreSinkWrite.Provider storeSinkWriteProvider;
+ private final String initialCommitUser;
+ private final Catalog.Loader catalogLoader;
+ protected Catalog catalog;
+ protected FileStoreTable table;
+ protected Map<Identifier, FileStoreTable> tables;
+ protected StoreSinkWriteState state;
+ protected StoreSinkWrite write;
+ protected Map<Identifier, StoreSinkWrite> writes;
+ protected String commitUser;
+
+ public CdcRecordStoreMultiWriteOperator(
+ Catalog.Loader catalogLoader,
+ StoreSinkWrite.Provider storeSinkWriteProvider,
+ String initialCommitUser) {
+ this.catalogLoader = catalogLoader;
+ this.storeSinkWriteProvider = storeSinkWriteProvider;
+ this.initialCommitUser = initialCommitUser;
+ this.retrySleepMillis = RETRY_SLEEP_TIME.defaultValue().toMillis();
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ catalog = catalogLoader.load();
+
+ // Each job can only have one user name and this name must be consistent across restarts.
+ // We cannot use job id as commit user name here because user may change job id by creating
+ // a savepoint, stop the job and then resume from savepoint.
+ commitUser =
+ StateUtils.getSingleValueFromState(
+ context, "commit_user_state", String.class, initialCommitUser);
+
+ state = new StoreSinkWriteState(context, (tableName, partition, bucket) -> true);
Review Comment:
Here add a TODO, we should use `CdcRecordMultiChannelComputer` to filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203199143
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/MultiTableCommittable.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.catalog.Identifier;
+
+/**
+ * MultiTableCommittable produced by {@link PrepareCommitOperator}. This type of Committable will
+ * only be produced by multiplexed operators that handles data from multiple tables. Thus, the
+ * database, table, and commit user of each table is included in the committable.
+ */
+public class MultiTableCommittable extends Committable {
Review Comment:
The extending class is necessary here becase in the base case `PrepareCommitOperator`, we have the following signature:
```java
protected abstract List<Committable> prepareCommit(boolean doCompaction, long checkpointId)
throws IOException;
```
Therefore, the new operator should also produce a subclass of committable and use another custom serializer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203200871
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/cdc/MultiplexCdcRecord.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.types.RowKind;
+
+import java.util.Map;
+
+/**
+ * A data change message from the CDC source. Compared to {@link CdcRecord}, MultiplexCdcRecord
+ * contains database and table information.
+ */
+public class MultiplexCdcRecord extends CdcRecord {
Review Comment:
Yes a field is better than extending here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203386938
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/MultiTableCommittable.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.catalog.Identifier;
+
+/**
+ * MultiTableCommittable produced by {@link PrepareCommitOperator}. This type of Committable will
+ * only be produced by multiplexed operators that handles data from multiple tables. Thus, the
+ * database, table, and commit user of each table is included in the committable.
+ */
+public class MultiTableCommittable extends Committable {
Review Comment:
Done refactoring to PrepareCommitOperator<IN, OUT>. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1202402902
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Identifier.java:
##########
@@ -109,4 +109,8 @@ public int hashCode() {
public String toString() {
return "Identifier{" + "database='" + database + '\'' + ", table='" + table + '\'' + '}';
}
+
+ public String getTableName() {
Review Comment:
Put this together with `getObjectName`.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecordStoreMultiWriteOperator.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.flink.sink.Committable;
+import org.apache.paimon.flink.sink.MultiTableCommittable;
+import org.apache.paimon.flink.sink.PrepareCommitOperator;
+import org.apache.paimon.flink.sink.StateUtils;
+import org.apache.paimon.flink.sink.StoreSinkWrite;
+import org.apache.paimon.flink.sink.StoreSinkWriteState;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static org.apache.paimon.flink.sink.cdc.CdcRecordStoreWriteOperator.RETRY_SLEEP_TIME;
+
+/**
+ * A {@link PrepareCommitOperator} to write {@link CdcRecord}. Record schema may change. If current
+ * known schema does not fit record schema, this operator will wait for schema changes.
+ */
+public class CdcRecordStoreMultiWriteOperator extends PrepareCommitOperator<MultiplexCdcRecord> {
+
+ private static final long serialVersionUID = 1L;
+ private final Catalog catalog;
Review Comment:
It is better to create a `CatalogLoader` instead of making `Catalog` serializable.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/MultiTableCommittable.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.catalog.Identifier;
+
+/**
+ * MultiTableCommittable produced by {@link PrepareCommitOperator}. This type of Committable will
+ * only be produced by multiplexed operators that handles data from multiple tables. Thus, the
+ * database, table, and commit user of each table is included in the committable.
+ */
+public class MultiTableCommittable extends Committable {
+
+ private final String database;
+ private final String table;
+ private final String commitUser;
+
+ public MultiTableCommittable(
+ String database,
+ String table,
+ String commitUser,
Review Comment:
Do we need this one? We can set same UUID user `StreamWriteBuilder.withCommitUser` to writer and committer.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/cdc/MultiplexCdcRecord.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.types.RowKind;
+
+import java.util.Map;
+
+/**
+ * A data change message from the CDC source. Compared to {@link CdcRecord}, MultiplexCdcRecord
+ * contains database and table information.
+ */
+public class MultiplexCdcRecord extends CdcRecord {
Review Comment:
Can we use just create a field `CdcRecord ` instead of extending?
Extending will make the caller think it can be used as a CdcRecord.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/MultiTableCommittable.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.catalog.Identifier;
+
+/**
+ * MultiTableCommittable produced by {@link PrepareCommitOperator}. This type of Committable will
+ * only be produced by multiplexed operators that handles data from multiple tables. Thus, the
+ * database, table, and commit user of each table is included in the committable.
+ */
+public class MultiTableCommittable extends Committable {
Review Comment:
Can we use just create a field `Committable` instead of extending?
Extending will make the caller think it can be used as a Committable.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecordStoreMultiWriteOperator.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.flink.sink.Committable;
+import org.apache.paimon.flink.sink.MultiTableCommittable;
+import org.apache.paimon.flink.sink.PrepareCommitOperator;
+import org.apache.paimon.flink.sink.StateUtils;
+import org.apache.paimon.flink.sink.StoreSinkWrite;
+import org.apache.paimon.flink.sink.StoreSinkWriteState;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static org.apache.paimon.flink.sink.cdc.CdcRecordStoreWriteOperator.RETRY_SLEEP_TIME;
+
+/**
+ * A {@link PrepareCommitOperator} to write {@link CdcRecord}. Record schema may change. If current
+ * known schema does not fit record schema, this operator will wait for schema changes.
+ */
+public class CdcRecordStoreMultiWriteOperator extends PrepareCommitOperator<MultiplexCdcRecord> {
+
+ private static final long serialVersionUID = 1L;
+ private final Catalog catalog;
+ private final StoreSinkWrite.Provider storeSinkWriteProvider;
+ private final String initialCommitUser;
+ private FileStoreTable table;
+ private Map<Identifier, FileStoreTable> tables;
+ private StoreSinkWriteState state;
+ private StoreSinkWrite write;
+ private Map<Identifier, StoreSinkWrite> writes;
+ private String commitUser;
+
+ public CdcRecordStoreMultiWriteOperator(
+ Catalog catalog,
+ StoreSinkWrite.Provider storeSinkWriteProvider,
+ String initialCommitUser) {
+ this.catalog = catalog;
+ this.storeSinkWriteProvider = storeSinkWriteProvider;
+ this.initialCommitUser = initialCommitUser;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Each job can only have one user name and this name must be consistent across restarts.
+ // We cannot use job id as commit user name here because user may change job id by creating
+ // a savepoint, stop the job and then resume from savepoint.
+ commitUser =
+ StateUtils.getSingleValueFromState(
+ context, "commit_user_state", String.class, initialCommitUser);
+
+ state = new StoreSinkWriteState(context, (tableName, partition, bucket) -> true);
+ this.tables = new HashMap<>();
+ this.writes = new HashMap<>();
+ }
+
+ @Override
+ public void processElement(StreamRecord<MultiplexCdcRecord> element) throws Exception {
+ MultiplexCdcRecord record = element.getValue();
+
+ String databaseName = record.getDatabaseName();
+ String tableName = record.getTableName();
+ Identifier key = Identifier.create(databaseName, tableName);
+ table =
+ tables.computeIfAbsent(
+ key,
+ id -> {
+ try {
+ return (FileStoreTable) catalog.getTable(id);
+ } catch (Catalog.TableNotExistException e) {
+ return null;
+ }
+ });
+
+ if (table == null) {
Review Comment:
here should be a `while (true) {...}` to wait table creation by `UpdatedDataFieldsProcessFunction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203330091
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/MultiTableCommittable.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.catalog.Identifier;
+
+/**
+ * MultiTableCommittable produced by {@link PrepareCommitOperator}. This type of Committable will
+ * only be produced by multiplexed operators that handles data from multiple tables. Thus, the
+ * database, table, and commit user of each table is included in the committable.
+ */
+public class MultiTableCommittable extends Committable {
Review Comment:
I think it is OK to bring generic type to `PrepareCommitOperator`.
`PrepareCommitOperator<IN, OUT>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1203361518
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecordStoreMultiWriteOperator.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.flink.sink.Committable;
+import org.apache.paimon.flink.sink.MultiTableCommittable;
+import org.apache.paimon.flink.sink.PrepareCommitOperator;
+import org.apache.paimon.flink.sink.StateUtils;
+import org.apache.paimon.flink.sink.StoreSinkWrite;
+import org.apache.paimon.flink.sink.StoreSinkWriteState;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static org.apache.paimon.flink.sink.cdc.CdcRecordStoreWriteOperator.RETRY_SLEEP_TIME;
+
+/**
+ * A {@link PrepareCommitOperator} to write {@link CdcRecord}. Record schema may change. If current
+ * known schema does not fit record schema, this operator will wait for schema changes.
+ */
+public class CdcRecordStoreMultiWriteOperator extends PrepareCommitOperator<MultiplexCdcRecord> {
+
+ private static final long serialVersionUID = 1L;
+ private final Catalog catalog;
+ private final StoreSinkWrite.Provider storeSinkWriteProvider;
+ private final String initialCommitUser;
+ private FileStoreTable table;
+ private Map<Identifier, FileStoreTable> tables;
+ private StoreSinkWriteState state;
+ private StoreSinkWrite write;
+ private Map<Identifier, StoreSinkWrite> writes;
+ private String commitUser;
+
+ public CdcRecordStoreMultiWriteOperator(
+ Catalog catalog,
+ StoreSinkWrite.Provider storeSinkWriteProvider,
+ String initialCommitUser) {
+ this.catalog = catalog;
+ this.storeSinkWriteProvider = storeSinkWriteProvider;
+ this.initialCommitUser = initialCommitUser;
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ // Each job can only have one user name and this name must be consistent across restarts.
+ // We cannot use job id as commit user name here because user may change job id by creating
+ // a savepoint, stop the job and then resume from savepoint.
+ commitUser =
+ StateUtils.getSingleValueFromState(
+ context, "commit_user_state", String.class, initialCommitUser);
+
+ state = new StoreSinkWriteState(context, (tableName, partition, bucket) -> true);
+ this.tables = new HashMap<>();
+ this.writes = new HashMap<>();
+ }
+
+ @Override
+ public void processElement(StreamRecord<MultiplexCdcRecord> element) throws Exception {
+ MultiplexCdcRecord record = element.getValue();
+
+ String databaseName = record.getDatabaseName();
+ String tableName = record.getTableName();
+ Identifier key = Identifier.create(databaseName, tableName);
+ table =
+ tables.computeIfAbsent(
+ key,
+ id -> {
+ try {
+ return (FileStoreTable) catalog.getTable(id);
+ } catch (Catalog.TableNotExistException e) {
+ return null;
+ }
+ });
+
+ if (table == null) {
Review Comment:
That makes sense...
Changed the code and added async table create test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1204918095
##########
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/sink/cdc/CdcRecordStoreMultiWriteOperatorTest.java:
##########
@@ -0,0 +1,696 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink.cdc;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.flink.sink.MultiTableCommittable;
+import org.apache.paimon.flink.sink.MultiTableCommittableTypeInfo;
+import org.apache.paimon.flink.sink.StoreSinkWriteImpl;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.schema.Schema;
+import org.apache.paimon.schema.SchemaChange;
+import org.apache.paimon.schema.SchemaManager;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypes;
+import org.apache.paimon.types.RowKind;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.TraceableFileIO;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.JavaSerializer;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Predicate;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link CdcRecordStoreMultiWriteOperator}. */
+public class CdcRecordStoreMultiWriteOperatorTest {
+
+ @TempDir java.nio.file.Path tempDir;
+
+ private String commitUser;
+ private Path warehouse;
+ private String databaseName;
+ private Identifier firstTable;
+ private Catalog catalog;
+ private Identifier secondTable;
+ private Catalog.Loader catalogLoader;
+ private Schema firstTableSchema;
+
+ @BeforeEach
+ public void before() throws Exception {
+ warehouse = new Path(TraceableFileIO.SCHEME + "://" + tempDir.toString());
+ commitUser = UUID.randomUUID().toString();
+
+ databaseName = "test_db";
+ firstTable = Identifier.create(databaseName, "test_table1");
+ secondTable = Identifier.create(databaseName, "test_table2");
+
+ catalogLoader = createCatalogLoader();
+ catalog = catalogLoader.load();
+ catalog.createDatabase(databaseName, true);
+ Options conf = new Options();
+ conf.set(CdcRecordStoreWriteOperator.RETRY_SLEEP_TIME, Duration.ofMillis(10));
+
+ RowType rowType1 =
+ RowType.of(
+ new DataType[] {DataTypes.INT(), DataTypes.INT(), DataTypes.STRING()},
+ new String[] {"pt", "k", "v"});
+ RowType rowType2 =
+ RowType.of(
+ new DataType[] {
+ DataTypes.INT(),
+ DataTypes.INT(),
+ DataTypes.FLOAT(),
+ DataTypes.VARCHAR(5),
+ DataTypes.VARBINARY(5)
+ },
+ new String[] {"k", "v1", "v2", "v3", "v4"});
+
+ firstTableSchema =
+ new Schema(
+ rowType1.getFields(),
+ Collections.singletonList("pt"),
+ Arrays.asList("pt", "k"),
+ conf.toMap(),
+ "");
+ createTestTables(
+ catalog,
+ Tuple2.of(firstTable, firstTableSchema),
+ Tuple2.of(
+ secondTable,
+ new Schema(
+ rowType2.getFields(),
+ Collections.emptyList(),
+ Collections.singletonList("k"),
+ conf.toMap(),
+ "")));
+ }
+
+ private void createTestTables(Catalog catalog, Tuple2<Identifier, Schema>... tableSpecs)
+ throws Exception {
+
+ for (Tuple2<Identifier, Schema> spec : tableSpecs) {
+ catalog.createTable(spec.f0, spec.f1, false);
+ }
+ }
+
+ @AfterEach
+ public void after() {
+ // assert all connections are closed
+ Predicate<Path> pathPredicate = path -> path.toString().contains(tempDir.toString());
+ assertThat(TraceableFileIO.openInputStreams(pathPredicate)).isEmpty();
+ assertThat(TraceableFileIO.openOutputStreams(pathPredicate)).isEmpty();
+ }
+
+ private static FileIO getFileIO(CatalogContext catalogContext, Path warehouse) {
+ FileIO fileIO;
+ try {
+ fileIO = FileIO.get(warehouse, catalogContext);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ return fileIO;
+ }
+
+ @Test
+ @Timeout(30)
+ public void testAsyncTableCreate() throws Exception {
+ // the async table will have same row type, partitions, and pks as firstTable
+ Identifier tableId = Identifier.create(databaseName, "async_new_table");
+
+ OneInputStreamOperatorTestHarness<CdcMultiplexRecord, MultiTableCommittable> harness =
+ createTestHarness(catalogLoader);
+
+ harness.open();
+
+ Runner runner = new Runner(harness);
+ Thread t = new Thread(runner);
+ t.start();
+
+ // check that records should be processed after table is created
+ Map<String, String> fields = new HashMap<>();
+ fields.put("pt", "0");
+ fields.put("k", "1");
+ fields.put("v", "10");
+
+ CdcMultiplexRecord expected =
+ CdcMultiplexRecord.fromCdcRecord(
+ databaseName,
+ tableId.getObjectName(),
+ new CdcRecord(RowKind.INSERT, fields));
+ runner.offer(expected);
+ CdcMultiplexRecord actual = runner.poll(1);
+
+ assertThat(actual).isNull();
+
+ catalog.createTable(tableId, firstTableSchema, true);
+ actual = runner.take();
+ assertThat(actual).isEqualTo(expected);
+
+ // after table is created, record should be processed immediately
+ fields = new HashMap<>();
+ fields.put("pt", "0");
+ fields.put("k", "3");
+ fields.put("v", "30");
+ expected =
+ CdcMultiplexRecord.fromCdcRecord(
+ databaseName,
+ tableId.getObjectName(),
+ new CdcRecord(RowKind.INSERT, fields));
+ runner.offer(expected);
+ actual = runner.take();
+ assertThat(actual).isEqualTo(expected);
+
+ runner.stop();
+ t.join();
+ harness.close();
+ }
+
+ @Test
+ @Timeout(30)
+ public void testInitializeState() throws Exception {
Review Comment:
I mean using `OneInputStreamOperatorTestHarness.initializeState` to test state recover for the operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1212: [feature] Support multi-table write operator
Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on code in PR #1212:
URL: https://github.com/apache/incubator-paimon/pull/1212#discussion_r1204042450
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/CommittableType.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+/** CommittableType produced by {@link PrepareCommitOperator}. */
+public interface CommittableType {
Review Comment:
No we don't :) Deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org