You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "Yizhou-Yang (via GitHub)" <gi...@apache.org> on 2023/06/09 08:42:12 UTC
[GitHub] [inlong] Yizhou-Yang opened a new pull request, #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Yizhou-Yang opened a new pull request, #8209:
URL: https://github.com/apache/inlong/pull/8209
### Prepare a Pull Request
- Fixes #7900
### Motivation
See issue.
### Modifications
Added PrimaryKeyPartitioner, and made modifications to support upsert-kafka
### Verifying this change
1. execute the sql in the issue again
2. create a kafka-mysql task to consume all the messages back to mysql
the mysql table fluctuated from 1000-2000 rows, and in the end it reduced to 0 rows, so I assume that the issue has been solved, since if it is still round robin, some update queries will be unordered, and it is unlikely that 3890 remaining records can be reduced to 0. The format used is json, for canal-json/avro/csv, I only tested that the primaryKey partitition is in effect and there are no errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1250087597
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/partitioner/SingleTablePrimaryKeyPartitioner.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka;
+
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.RowData.FieldGetter;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * The PrimaryKey Partitioner is used to extract primary key from single table messages:
Review Comment:
typo here extra :
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1255464410
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/table/UpsertKafkaDynamicTableFactory.java:
##########
@@ -274,12 +274,12 @@ private Optional<FlinkKafkaPartitioner<RowData>> getFlinkKafkaPartitioner(
ReadableConfig tableOptions, ClassLoader classLoader, TableSchema schema) {
if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
&& SINK_PARTITIONER_VALUE_PRIMARY_KEY.equals(tableOptions.getOptional(SINK_PARTITIONER).get())) {
- SingleTablePrimaryKeyPartitioner<RowData> primaryKeyPartitioner = new SingleTablePrimaryKeyPartitioner<>();
- primaryKeyPartitioner.setPartitionNumber(tableOptions.getOptional(SINK_FIXED_IDENTIFIER).orElse(null));
- primaryKeyPartitioner.setPartitionKey(tableOptions.getOptional(SINK_MULTIPLE_PARTITION_PATTERN)
+ SingleTableCustomFieldsPartitioner<RowData> customFIeldsPartitioner = new SingleTableCustomFieldsPartitioner<>();
Review Comment:
customFIeldsPartitioner -> customFieldsPartitioner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1250067203
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/partitioner/SingleTablePrimaryKeyPartitioner.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka;
+
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.RowData.FieldGetter;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * The PrimaryKey Partitioner is used to extract primary key from single table messages:
Review Comment:
typo here , extra :
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "Yizhou-Yang (via GitHub)" <gi...@apache.org>.
Yizhou-Yang commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1257642555
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/table/UpsertKafkaDynamicTableFactory.java:
##########
@@ -96,6 +102,17 @@ public class UpsertKafkaDynamicTableFactory
DynamicTableSinkFactory {
public static final String IDENTIFIER = "upsert-kafka-inlong";
+ public static final ConfigOption<String> SINK_MULTIPLE_PARTITION_PATTERN =
+ ConfigOptions.key("sink.multiple.partition-pattern")
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ "option 'sink.multiple.partition-pattern' used when the partitioner is raw-hash， and used to pass in the primary key partition in primary key partitioners");;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "Yizhou-Yang (via GitHub)" <gi...@apache.org>.
Yizhou-Yang commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1254326753
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/table/KafkaDynamicTableFactory.java:
##########
@@ -260,15 +268,23 @@ private Optional<FlinkKafkaPartitioner<RowData>> getFlinkKafkaPartitioner(
inLongFixedPartitionPartitioner.setSinkMultipleFormat(tableOptions.getOptional(SINK_MULTIPLE_FORMAT)
.orElse(null));
return Optional.of(inLongFixedPartitionPartitioner);
- }
- if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
+ } else if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
&& SINK_PARTITIONER_VALUE_RAW_HASH.equals(tableOptions.getOptional(SINK_PARTITIONER).get())) {
RawDataHashPartitioner<RowData> rawHashPartitioner = new RawDataHashPartitioner<>(tableOptions
.getOptional(DATASOURCE_PARTITION_MAP).orElse(Collections.emptyMap()));
rawHashPartitioner.setSinkMultipleFormat(tableOptions.getOptional(SINK_MULTIPLE_FORMAT).orElse(null));
rawHashPartitioner.setPartitionPattern(tableOptions.getOptional(SINK_MULTIPLE_PARTITION_PATTERN)
.orElse(null));
return Optional.of(rawHashPartitioner);
+ } else if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
+ && SINK_PARTITIONER_VALUE_PRIMARY_KEY.equals(tableOptions.getOptional(SINK_PARTITIONER).get())) {
+ SingleTablePrimaryKeyPartitioner<RowData> primaryKeyPartitioner = new SingleTablePrimaryKeyPartitioner<>();
+ // the pattern is different from the ${} pattern, it is a truncated string of schema fields.
+ primaryKeyPartitioner.setPartitionNumber(tableOptions.getOptional(SINK_FIXED_IDENTIFIER).orElse(null));
+ primaryKeyPartitioner.setPartitionKey(tableOptions.getOptional(SINK_MULTIPLE_PARTITION_PATTERN)
Review Comment:
where should I update the document? I can't find any description..
<img width="1766" alt="image" src="https://github.com/apache/inlong/assets/32808678/f5f4b243-9566-4163-a2b8-3bbe54dc9fef">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1255474145
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/partitioner/SingleTablePrimaryKeyPartitioner.java:
##########
Review Comment:
your class file name is error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang merged PR #8209:
URL: https://github.com/apache/inlong/pull/8209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1250111812
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/test/java/org/apache/inlong/sort/kafka/partitioner/SingleTablePrimaryKeyPartitionerTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka.partitioner;
+
+import org.apache.inlong.sort.base.format.AbstractDynamicSchemaFormat;
+import org.apache.inlong.sort.kafka.SingleTablePrimaryKeyPartitioner;
+
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.types.RowKind;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * The unit tests for {@link
+ * org.apache.inlong.sort.kafka.partitioner.SingleTablePrimaryKeyPartitioner}.
+ */
+public class SingleTablePrimaryKeyPartitionerTest {
+
+ private String databasePattern = "${database}";
+ private String tablePattern = "${table}";
+ private AbstractDynamicSchemaFormat dynamicSchemaFormat = new TestDynamicSchemaFormat();
+ class TestDynamicSchemaFormat extends AbstractDynamicSchemaFormat {
+
+ @Override
+ public String extract(Object data, String key) {
+ return null;
+ }
+
+ @Override
+ public List<String> extractPrimaryKeyNames(Object data) {
+ return null;
+ }
+
+ @Override
+ public boolean extractDDLFlag(Object data) {
+ return false;
+ }
+
+ @Override
+ public String extractDDL(Object data) {
+ return null;
+ }
+
+ @Override
+ public Object extractOperation(Object data) {
+ return null;
+ }
+
+ @Override
+ public List<RowData> extractRowData(Object data, RowType rowType) {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(byte[] message) throws IOException {
+ return null;
+ }
+
+ @Override
+ public String parse(Object data, String pattern) throws IOException {
+ if (databasePattern.equals(pattern))
+ return "db1";
+ if (tablePattern.equals(pattern))
+ return "tb1";
+ return null;
+ }
+
+ @Override
+ public RowType extractSchema(Object data, List pkNames) {
+ return null;
+ }
+ }
+
+ @Test
+ public void testPrimaryKeyPartitioner() {
+ SingleTablePrimaryKeyPartitioner singleTablePrimaryKeyPartitioner =
+ new SingleTablePrimaryKeyPartitioner();
+ RowData record = new GenericRowData(2);
+ RowKind kind = RowKind.fromByteValue((byte) 0);
+ record.setRowKind(kind);
+ int partition = singleTablePrimaryKeyPartitioner.partition(record, null,
+ null, null, new int[]{0, 1, 2, 3});
+ Assert.assertEquals(0, partition);
+ }
Review Comment:
This test code can't test your logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on PR #8209:
URL: https://github.com/apache/inlong/pull/8209#issuecomment-1624801331
I don't think this is primary key partition. It is partitioned by fixed field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "Yizhou-Yang (via GitHub)" <gi...@apache.org>.
Yizhou-Yang commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1255179997
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/test/java/org/apache/inlong/sort/kafka/partitioner/SingleTablePrimaryKeyPartitionerTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka.partitioner;
+
+import org.apache.inlong.sort.base.format.AbstractDynamicSchemaFormat;
+import org.apache.inlong.sort.kafka.SingleTablePrimaryKeyPartitioner;
+
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.types.RowKind;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * The unit tests for {@link
+ * org.apache.inlong.sort.kafka.partitioner.SingleTablePrimaryKeyPartitioner}.
+ */
+public class SingleTablePrimaryKeyPartitionerTest {
+
+ private String databasePattern = "${database}";
+ private String tablePattern = "${table}";
+ private AbstractDynamicSchemaFormat dynamicSchemaFormat = new TestDynamicSchemaFormat();
+ class TestDynamicSchemaFormat extends AbstractDynamicSchemaFormat {
+
+ @Override
+ public String extract(Object data, String key) {
+ return null;
+ }
+
+ @Override
+ public List<String> extractPrimaryKeyNames(Object data) {
+ return null;
+ }
+
+ @Override
+ public boolean extractDDLFlag(Object data) {
+ return false;
+ }
+
+ @Override
+ public String extractDDL(Object data) {
+ return null;
+ }
+
+ @Override
+ public Object extractOperation(Object data) {
+ return null;
+ }
+
+ @Override
+ public List<RowData> extractRowData(Object data, RowType rowType) {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(byte[] message) throws IOException {
+ return null;
+ }
+
+ @Override
+ public String parse(Object data, String pattern) throws IOException {
+ if (databasePattern.equals(pattern))
+ return "db1";
+ if (tablePattern.equals(pattern))
+ return "tb1";
+ return null;
+ }
+
+ @Override
+ public RowType extractSchema(Object data, List pkNames) {
+ return null;
+ }
+ }
+
+ @Test
+ public void testPrimaryKeyPartitioner() {
+ SingleTablePrimaryKeyPartitioner singleTablePrimaryKeyPartitioner =
+ new SingleTablePrimaryKeyPartitioner();
+ RowData record = new GenericRowData(2);
+ RowKind kind = RowKind.fromByteValue((byte) 0);
+ record.setRowKind(kind);
+ int partition = singleTablePrimaryKeyPartitioner.partition(record, null,
+ null, null, new int[]{0, 1, 2, 3});
+ Assert.assertEquals(0, partition);
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "Yizhou-Yang (via GitHub)" <gi...@apache.org>.
Yizhou-Yang commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1254327615
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/test/java/org/apache/inlong/sort/kafka/partitioner/SingleTablePrimaryKeyPartitionerTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka.partitioner;
+
+import org.apache.inlong.sort.base.format.AbstractDynamicSchemaFormat;
+import org.apache.inlong.sort.kafka.SingleTablePrimaryKeyPartitioner;
+
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.types.RowKind;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * The unit tests for {@link
+ * org.apache.inlong.sort.kafka.partitioner.SingleTablePrimaryKeyPartitioner}.
+ */
+public class SingleTablePrimaryKeyPartitionerTest {
+
+ private String databasePattern = "${database}";
+ private String tablePattern = "${table}";
+ private AbstractDynamicSchemaFormat dynamicSchemaFormat = new TestDynamicSchemaFormat();
+ class TestDynamicSchemaFormat extends AbstractDynamicSchemaFormat {
+
+ @Override
+ public String extract(Object data, String key) {
+ return null;
+ }
+
+ @Override
+ public List<String> extractPrimaryKeyNames(Object data) {
+ return null;
+ }
+
+ @Override
+ public boolean extractDDLFlag(Object data) {
+ return false;
+ }
+
+ @Override
+ public String extractDDL(Object data) {
+ return null;
+ }
+
+ @Override
+ public Object extractOperation(Object data) {
+ return null;
+ }
+
+ @Override
+ public List<RowData> extractRowData(Object data, RowType rowType) {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(byte[] message) throws IOException {
+ return null;
+ }
+
+ @Override
+ public String parse(Object data, String pattern) throws IOException {
+ if (databasePattern.equals(pattern))
+ return "db1";
+ if (tablePattern.equals(pattern))
+ return "tb1";
+ return null;
+ }
+
+ @Override
+ public RowType extractSchema(Object data, List pkNames) {
+ return null;
+ }
+ }
+
+ @Test
+ public void testPrimaryKeyPartitioner() {
+ SingleTablePrimaryKeyPartitioner singleTablePrimaryKeyPartitioner =
+ new SingleTablePrimaryKeyPartitioner();
+ RowData record = new GenericRowData(2);
+ RowKind kind = RowKind.fromByteValue((byte) 0);
+ record.setRowKind(kind);
+ int partition = singleTablePrimaryKeyPartitioner.partition(record, null,
+ null, null, new int[]{0, 1, 2, 3});
+ Assert.assertEquals(0, partition);
+ }
Review Comment:
sure , refactoring, might take some time to actually create the schemas and fieldgetters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on PR #8209:
URL: https://github.com/apache/inlong/pull/8209#issuecomment-1654943269
git pull upstream master -r
git rebase master
git push your branch
Don't modify code style by mistake to trigger CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] hnrainll commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "hnrainll (via GitHub)" <gi...@apache.org>.
hnrainll commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1284652164
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/partitioner/SingleTableCustomFieldsPartitioner.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka;
Review Comment:
Package name 'org.apache.inlong.sort.kafka' does not correspond to the file path 'org.apache.inlong.sort.kafka.partitioner'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] hnrainll commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "hnrainll (via GitHub)" <gi...@apache.org>.
hnrainll commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1285346839
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/partitioner/SingleTableCustomFieldsPartitioner.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka;
Review Comment:
No problem.
I have created issue: https://github.com/apache/inlong/issues/8641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "Yizhou-Yang (via GitHub)" <gi...@apache.org>.
Yizhou-Yang commented on PR #8209:
URL: https://github.com/apache/inlong/pull/8209#issuecomment-1600006836
> Please add the corresponding unit test. @Yizhou-Yang
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1257666518
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/test/java/org/apache/inlong/sort/kafka/partitioner/SingleTableCustomFieldsPartitionerTest.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka.partitioner;
+
+import org.apache.inlong.sort.kafka.SingleTableCustomFieldsPartitioner;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.RowData.FieldGetter;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.data.writer.BinaryRowWriter;
+import org.apache.flink.table.types.logical.IntType;
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * The unit tests for {@link
+ * org.apache.inlong.sort.kafka.partitioner.SingleTableCustomFieldsPartitioner}.
+ */
+public class SingleTableCustomFieldsPartitionerTest {
+
+ @Test
+ public void testPrimaryKeyPartitioner() {
+ SingleTableCustomFieldsPartitioner singleTablePrimaryKeyPartitioner =
Review Comment:
`singleTablePrimaryKeyPartitioner` is error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1255472260
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/table/UpsertKafkaDynamicTableFactory.java:
##########
@@ -96,6 +102,17 @@ public class UpsertKafkaDynamicTableFactory
DynamicTableSinkFactory {
public static final String IDENTIFIER = "upsert-kafka-inlong";
+ public static final ConfigOption<String> SINK_MULTIPLE_PARTITION_PATTERN =
+ ConfigOptions.key("sink.multiple.partition-pattern")
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ "option 'sink.multiple.partition-pattern' used when the partitioner is raw-hash， and used to pass in the primary key partition in primary key partitioners");;
Review Comment:
description need modify
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1257666939
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/table/UpsertKafkaDynamicTableFactory.java:
##########
@@ -96,6 +102,17 @@ public class UpsertKafkaDynamicTableFactory
DynamicTableSinkFactory {
public static final String IDENTIFIER = "upsert-kafka-inlong";
+ public static final ConfigOption<String> SINK_MULTIPLE_PARTITION_PATTERN =
+ ConfigOptions.key("sink.multiple.partition-pattern")
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ "option 'sink.multiple.partition-pattern' used when the partitioner is raw-hash， and used to pass in the primary key partition in primary key partitioners");;
Review Comment:
`used to pass in the primary key partition in primary key partitioners` is error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1250067203
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/partitioner/SingleTablePrimaryKeyPartitioner.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka;
+
+import org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.RowData.FieldGetter;
+import org.apache.flink.util.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * The PrimaryKey Partitioner is used to extract primary key from single table messages:
Review Comment:
typo here , extra :
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on PR #8209:
URL: https://github.com/apache/inlong/pull/8209#issuecomment-1588616261
@lordcheng10 please review this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on PR #8209:
URL: https://github.com/apache/inlong/pull/8209#issuecomment-1594032176
Please add the corresponding unit test.
@Yizhou-Yang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1250099726
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/test/java/org/apache/inlong/sort/kafka/partitioner/SingleTablePrimaryKeyPartitionerTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka.partitioner;
+
+import org.apache.inlong.sort.base.format.AbstractDynamicSchemaFormat;
+import org.apache.inlong.sort.kafka.SingleTablePrimaryKeyPartitioner;
+
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.types.RowKind;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * The unit tests for {@link
+ * org.apache.inlong.sort.kafka.partitioner.SingleTablePrimaryKeyPartitioner}.
+ */
+public class SingleTablePrimaryKeyPartitionerTest {
+
+ private String databasePattern = "${database}";
+ private String tablePattern = "${table}";
+ private AbstractDynamicSchemaFormat dynamicSchemaFormat = new TestDynamicSchemaFormat();
+ class TestDynamicSchemaFormat extends AbstractDynamicSchemaFormat {
+
+ @Override
+ public String extract(Object data, String key) {
+ return null;
+ }
+
+ @Override
+ public List<String> extractPrimaryKeyNames(Object data) {
+ return null;
+ }
+
+ @Override
+ public boolean extractDDLFlag(Object data) {
+ return false;
+ }
+
+ @Override
+ public String extractDDL(Object data) {
+ return null;
+ }
+
+ @Override
+ public Object extractOperation(Object data) {
+ return null;
+ }
+
+ @Override
Review Comment:
`TestDynamicSchemaFormat` is not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1250124633
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/table/KafkaDynamicTableFactory.java:
##########
@@ -260,15 +268,23 @@ private Optional<FlinkKafkaPartitioner<RowData>> getFlinkKafkaPartitioner(
inLongFixedPartitionPartitioner.setSinkMultipleFormat(tableOptions.getOptional(SINK_MULTIPLE_FORMAT)
.orElse(null));
return Optional.of(inLongFixedPartitionPartitioner);
- }
- if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
+ } else if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
&& SINK_PARTITIONER_VALUE_RAW_HASH.equals(tableOptions.getOptional(SINK_PARTITIONER).get())) {
RawDataHashPartitioner<RowData> rawHashPartitioner = new RawDataHashPartitioner<>(tableOptions
.getOptional(DATASOURCE_PARTITION_MAP).orElse(Collections.emptyMap()));
rawHashPartitioner.setSinkMultipleFormat(tableOptions.getOptional(SINK_MULTIPLE_FORMAT).orElse(null));
rawHashPartitioner.setPartitionPattern(tableOptions.getOptional(SINK_MULTIPLE_PARTITION_PATTERN)
.orElse(null));
return Optional.of(rawHashPartitioner);
+ } else if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
+ && SINK_PARTITIONER_VALUE_PRIMARY_KEY.equals(tableOptions.getOptional(SINK_PARTITIONER).get())) {
+ SingleTablePrimaryKeyPartitioner<RowData> primaryKeyPartitioner = new SingleTablePrimaryKeyPartitioner<>();
+ // the pattern is different from the ${} pattern, it is a truncated string of schema fields.
+ primaryKeyPartitioner.setPartitionNumber(tableOptions.getOptional(SINK_FIXED_IDENTIFIER).orElse(null));
+ primaryKeyPartitioner.setPartitionKey(tableOptions.getOptional(SINK_MULTIPLE_PARTITION_PATTERN)
Review Comment:
you reuse SINK_MULTIPLE_PARTITION_PATTERN to parse partition, pattern define of SINK_MULTIPLE_PARTITION_PATTERN is different with your define. I suggest add description in `withDescription`.
```java
public static final ConfigOption<String> SINK_MULTIPLE_PARTITION_PATTERN =
ConfigOptions.key("sink.multiple.partition-pattern")
.stringType()
.noDefaultValue()
.withDescription("option 'sink.multiple.partition-pattern' used when the partitioner is raw-hash.");
```
And update inlong document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "Yizhou-Yang (via GitHub)" <gi...@apache.org>.
Yizhou-Yang commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1255181201
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/table/KafkaDynamicTableFactory.java:
##########
@@ -260,15 +268,23 @@ private Optional<FlinkKafkaPartitioner<RowData>> getFlinkKafkaPartitioner(
inLongFixedPartitionPartitioner.setSinkMultipleFormat(tableOptions.getOptional(SINK_MULTIPLE_FORMAT)
.orElse(null));
return Optional.of(inLongFixedPartitionPartitioner);
- }
- if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
+ } else if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
&& SINK_PARTITIONER_VALUE_RAW_HASH.equals(tableOptions.getOptional(SINK_PARTITIONER).get())) {
RawDataHashPartitioner<RowData> rawHashPartitioner = new RawDataHashPartitioner<>(tableOptions
.getOptional(DATASOURCE_PARTITION_MAP).orElse(Collections.emptyMap()));
rawHashPartitioner.setSinkMultipleFormat(tableOptions.getOptional(SINK_MULTIPLE_FORMAT).orElse(null));
rawHashPartitioner.setPartitionPattern(tableOptions.getOptional(SINK_MULTIPLE_PARTITION_PATTERN)
.orElse(null));
return Optional.of(rawHashPartitioner);
+ } else if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
+ && SINK_PARTITIONER_VALUE_PRIMARY_KEY.equals(tableOptions.getOptional(SINK_PARTITIONER).get())) {
+ SingleTablePrimaryKeyPartitioner<RowData> primaryKeyPartitioner = new SingleTablePrimaryKeyPartitioner<>();
+ // the pattern is different from the ${} pattern, it is a truncated string of schema fields.
+ primaryKeyPartitioner.setPartitionNumber(tableOptions.getOptional(SINK_FIXED_IDENTIFIER).orElse(null));
+ primaryKeyPartitioner.setPartitionKey(tableOptions.getOptional(SINK_MULTIPLE_PARTITION_PATTERN)
Review Comment:
description has been added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
Posted by "Yizhou-Yang (via GitHub)" <gi...@apache.org>.
Yizhou-Yang commented on PR #8209:
URL: https://github.com/apache/inlong/pull/8209#issuecomment-1624833286
> I don't think this is primary key partition. It is partitioned by fixed field.
very solid point... this is not primary key partitioning, because the user needs to be able to select which fields they need to partition by. Needs to change the naming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1285311378
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/partitioner/SingleTableCustomFieldsPartitioner.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.kafka;
Review Comment:
@hnrainll Could you please create an issue and submit a PR to resolve it? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8209: [INLONG-7900][Sort] Support partition by custom fields when upsert single table of Kafka
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8209:
URL: https://github.com/apache/inlong/pull/8209#discussion_r1255445744
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/kafka/src/main/java/org/apache/inlong/sort/kafka/table/KafkaDynamicTableFactory.java:
##########
@@ -280,13 +280,13 @@ private Optional<FlinkKafkaPartitioner<RowData>> getFlinkKafkaPartitioner(
return Optional.of(rawHashPartitioner);
} else if (tableOptions.getOptional(SINK_PARTITIONER).isPresent()
&& SINK_PARTITIONER_VALUE_PRIMARY_KEY.equals(tableOptions.getOptional(SINK_PARTITIONER).get())) {
- SingleTablePrimaryKeyPartitioner<RowData> primaryKeyPartitioner = new SingleTablePrimaryKeyPartitioner<>();
+ SingleTableCustomFieldsPartitioner<RowData> customFIeldsPartitioner = new SingleTableCustomFieldsPartitioner<>();
Review Comment:
customFIeldsPartitioner -》 customFieldsPartitioner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org