You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/08/02 11:42:40 UTC
[GitHub] [flink] fapaul opened a new pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
fapaul opened a new pull request #16676:
URL: https://github.com/apache/flink/pull/16676
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
*(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)*
## Brief change log
*(for example:)*
- *The TaskInfo is stored in the blob store on job creation time as a persistent artifact*
- *Deployments RPC transmits only the blob storage reference*
- *TaskManagers retrieve the TaskInfo from the blob cache*
## Verifying this change
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (100MB)*
- *Extended integration test for recovery after master (JobManager) failure*
- *Added test that validates that TaskInfo is transferred only once across recoveries*
- *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes / no / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
- The S3 file system connector: (yes / no / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
}, {
"hash" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"triggerType" : "PUSH"
}, {
"hash" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* e8c6ec06f2263f5704ccf597494c43a1bd0e5893 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664)
* 5f636910791eb5ddaa6b9b210c3a4b3230fc60bd UNKNOWN
* 055d0fd83c72f141a1bd2d8a4bc0e30bf637687f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684156074
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/TransactionalIdFactory.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.base.Splitter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+class TransactionalIdFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TransactionalIdFactory.class);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
Review comment:
I renamed it to `checkpointOffset` and I hope the docstring for the parameter explains enough to make it apparent it has nothing to do with the partition offset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684152339
##########
File path: flink-core/src/main/java/org/apache/flink/api/connector/sink/InitContextInitializationContextAdapter.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.connector.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import java.util.function.Function;
+
+/**
+ * Adapter between {@link Sink.InitContext} and {@link SerializationSchema.InitializationContext}.
+ */
+public class InitContextInitializationContextAdapter
Review comment:
> I wonder if this should be in flink-connector-base instead.
It can also live in `flink-connector-base` I just refrained from putting it there because there is no sink-specific code yet and all these adapters are currently also in core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af9dd2b341dfe555224f46d3e14a7a83c1b5db04 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a4a37916fdd2d49a7682b2d90da22e5ed398eae5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429)
* 25fb6f9a84ab137be93a04bfad41c4ab021a3b4b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r683265825
##########
File path: flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointTestBase.java
##########
@@ -165,35 +161,11 @@ protected File execute(UnalignedSettings settings) throws Exception {
miniCluster.after();
}
if (settings.generateCheckpoint) {
- return Files.find(checkpointDir.toPath(), 2, this::isCompletedCheckpoint)
- .max(Comparator.comparing(Path::toString))
- .map(Path::toFile)
- .orElseThrow(() -> new IllegalStateException("Cannot generate checkpoint"));
+ return TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
Review comment:
This is not easily possible because I cannot add the `TestUtils` dependency because it would introduce a cycling dependency. Therefore I left it, for now, to not spending more time entangling the dependencies.
##########
File path: flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointTestBase.java
##########
@@ -165,35 +161,11 @@ protected File execute(UnalignedSettings settings) throws Exception {
miniCluster.after();
}
if (settings.generateCheckpoint) {
- return Files.find(checkpointDir.toPath(), 2, this::isCompletedCheckpoint)
- .max(Comparator.comparing(Path::toString))
- .map(Path::toFile)
- .orElseThrow(() -> new IllegalStateException("Cannot generate checkpoint"));
+ return TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
Review comment:
This is not easily possible because I cannot add the `TestUtils` dependency because it would introduce a cycling dependency. Therefore I left it, for now, to not spending more time disentangling the dependencies.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/FlinkKafkaInternalProducer.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.TransactionalRequestResult;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Properties;
+
+/**
+ * A {@link KafkaProducer} that exposes private fields to allow resume producing from a given state.
+ */
+class FlinkKafkaInternalProducer<K, V> extends KafkaProducer<K, V> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkKafkaInternalProducer.class);
+ private static final String TRANSACTION_MANAGER_STATE_ENUM =
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State";
+
+ private final Properties kafkaProducerConfig;
+ @Nullable private final String transactionalId;
+
+ public FlinkKafkaInternalProducer(Properties properties) {
+ super(properties);
+ this.kafkaProducerConfig = properties;
+ this.transactionalId = properties.getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ if (transactionalId != null) {
+ flushNewPartitions();
+ }
+ }
+
+ public Properties getKafkaProducerConfig() {
+ return kafkaProducerConfig;
+ }
+
+ public short getEpoch() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (short) getField(producerIdAndEpoch, "epoch");
+ }
+
+ public long getProducerId() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (long) getField(producerIdAndEpoch, "producerId");
+ }
+
+ /**
+ * Besides committing {@link org.apache.kafka.clients.producer.KafkaProducer#commitTransaction}
+ * is also adding new partitions to the transaction. flushNewPartitions method is moving this
+ * logic to pre-commit/flush, to make resumeTransaction simpler. Otherwise resumeTransaction
+ * would require to restore state of the not yet added/"in-flight" partitions.
+ */
+ private void flushNewPartitions() {
+ LOG.info("Flushing new partitions");
+ TransactionalRequestResult result = enqueueNewPartitions();
+ Object sender = getField("sender");
+ invoke(sender, "wakeup");
+ result.await();
+ }
+
+ /**
+ * Enqueues new transactions at the transaction manager and returns a {@link
+ * TransactionalRequestResult} that allows waiting on them.
+ *
+ * <p>If there are no new transactions we return a {@link TransactionalRequestResult} that is
+ * already done.
+ */
+ private TransactionalRequestResult enqueueNewPartitions() {
+ Object transactionManager = getField("transactionManager");
+ synchronized (transactionManager) {
+ Object newPartitionsInTransaction =
+ getField(transactionManager, "newPartitionsInTransaction");
+ Object newPartitionsInTransactionIsEmpty =
+ invoke(newPartitionsInTransaction, "isEmpty");
+ TransactionalRequestResult result;
+ if (newPartitionsInTransactionIsEmpty instanceof Boolean
+ && !((Boolean) newPartitionsInTransactionIsEmpty)) {
+ Object txnRequestHandler =
+ invoke(transactionManager, "addPartitionsToTransactionHandler");
+ invoke(
+ transactionManager,
+ "enqueueRequest",
+ new Class[] {txnRequestHandler.getClass().getSuperclass()},
+ new Object[] {txnRequestHandler});
+ result =
+ (TransactionalRequestResult)
+ getField(
+ txnRequestHandler,
+ txnRequestHandler.getClass().getSuperclass(),
+ "result");
+ } else {
+ // we don't have an operation but this operation string is also used in
+ // addPartitionsToTransactionHandler.
+ result = new TransactionalRequestResult("AddPartitionsToTxn");
+ result.done();
+ }
+ return result;
+ }
+ }
+
+ private static Object invoke(Object object, String methodName, Object... args) {
+ Class<?>[] argTypes = new Class[args.length];
+ for (int i = 0; i < args.length; i++) {
+ argTypes[i] = args[i].getClass();
+ }
+ return invoke(object, methodName, argTypes, args);
+ }
+
+ private static Object invoke(
+ Object object, String methodName, Class<?>[] argTypes, Object[] args) {
+ try {
+ Method method = object.getClass().getDeclaredMethod(methodName, argTypes);
+ method.setAccessible(true);
+ return method.invoke(object, args);
+ } catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
Review comment:
Can you give an example of what exactly you want to see here?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
+ * #setDeliverGuarantee(DeliveryGuarantee)} but keep in mind when using {@link
+ * DeliveryGuarantee#EXACTLY_ONCE} one must set the transactionalIdPrefix {@link
+ * #setTransactionalIdPrefix(String)}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+public class KafkaSinkBuilder<IN> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkBuilder.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private String transactionalIdPrefix = "kafka-sink";
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+ private String bootstrapServers;
+
+ /**
+ * Sets the wanted the {@link DeliveryGuarantee}. The default delivery guarantee is {@link
+ * #deliveryGuarantee}.
+ *
+ * @param deliveryGuarantee
+ * @return {@link KafkaSinkBuilder}
+ */
+ KafkaSinkBuilder<IN> setDeliverGuarantee(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "semantic");
Review comment:
I will definitely leave the second parameter out next time but now I want to keep it consistent.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
+
+ private static byte readTransactionState(ByteBuffer buffer) {
+ // producerId
+ buffer.getLong();
+ // epoch
+ buffer.getShort();
+ // transactionTimeout
+ buffer.getInt();
+ // statusKey
+ return buffer.get();
+ }
+
+ enum TransactionState {
+ Empty(Byte.parseByte("0")),
Review comment:
I am not convinced by all options 😄 Do you mind if I leave it as is?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
+
+ private static byte readTransactionState(ByteBuffer buffer) {
+ // producerId
+ buffer.getLong();
+ // epoch
+ buffer.getShort();
+ // transactionTimeout
+ buffer.getInt();
+ // statusKey
+ return buffer.get();
+ }
Review comment:
Hmm, I am in slight favor to keep it like this to at least claim to know why this is the important byte. Do you think performance-wise it is necessary to do the point lookup?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
Review comment:
I like the current way of having it explicitly. The impact of having a different item at the head of the list is quite critical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684123450
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkITCase.java
##########
@@ -0,0 +1,634 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.StateBackendOptions;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.test.util.TestUtils;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.testutils.junit.SharedReference;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.CreateTopicsResult;
+import org.apache.kafka.clients.admin.DeleteTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.LongStream;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+/** Tests for using KafkaSink writing to a Kafka cluster. */
+public class KafkaSinkITCase extends TestLogger {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkITCase.class);
+ private static final Slf4jLogConsumer LOG_CONSUMER = new Slf4jLogConsumer(LOG);
+ private static final String INTER_CONTAINER_KAFKA_ALIAS = "kafka";
+ private static final Network NETWORK = Network.newNetwork();
+ private static final int ZK_TIMEOUT_MILLIS = 30000;
+ private static final short TOPIC_REPLICATION_FACTOR = 1;
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+
+ private String topic;
+ private SharedReference<AtomicLong> emittedRecordsCount;
+ private SharedReference<AtomicLong> emittedRecordsWithCheckpoint;
+ private SharedReference<AtomicBoolean> failed;
+ private SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ @ClassRule
+ public static final KafkaContainer KAFKA_CONTAINER =
+ new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.5.2"))
+ .withEmbeddedZookeeper()
+ .withEnv(
+ ImmutableMap.of(
+ "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR",
+ "1",
+ "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
+ String.valueOf(Duration.ofHours(2).toMillis()),
+ "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR",
+ "1",
+ "KAFKA_MIN_INSYNC_REPLICAS",
+ "1"))
+ .withNetwork(NETWORK)
+ .withLogConsumer(LOG_CONSUMER)
+ .withNetworkAliases(INTER_CONTAINER_KAFKA_ALIAS);
+
+ @Rule public final SharedObjects sharedObjects = SharedObjects.create();
+
+ @Rule public final TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void setUp() throws ExecutionException, InterruptedException, TimeoutException {
+ emittedRecordsCount = sharedObjects.add(new AtomicLong());
+ emittedRecordsWithCheckpoint = sharedObjects.add(new AtomicLong());
+ failed = sharedObjects.add(new AtomicBoolean(false));
+ lastCheckpointedRecord = sharedObjects.add(new AtomicLong(0));
+ topic = UUID.randomUUID().toString();
+ createTestTopic(topic, 1, TOPIC_REPLICATION_FACTOR);
+ }
+
+ @After
+ public void tearDown() throws ExecutionException, InterruptedException, TimeoutException {
+ deleteTestTopic(topic);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithAtLeastOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.AT_LEAST_ONCE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithNoneGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.NONE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithExactlyOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.EXACTLY_ONCE, emittedRecordsWithCheckpoint);
+ }
+
+ @Test
+ public void testRecoveryWithAtLeastOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.AT_LEAST_ONCE,
+ (records) ->
+ assertThat(records, hasItems(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L)));
+ }
+
+ @Test
+ public void testRecoveryWithExactlyOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.EXACTLY_ONCE,
+ (records) ->
+ assertEquals(
+ records,
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList())));
+ }
+
+ @Test
+ public void testAbortTransactionsOfPendingCheckpointsAfterFailure() throws Exception {
+ // Run a first job failing during the async phase of a checkpoint to leave some
+ // lingering transactions
+ final Configuration config = new Configuration();
+ config.setString(StateBackendOptions.STATE_BACKEND, "filesystem");
+ final File checkpointDir = temp.newFolder();
+ config.setString(
+ CheckpointingOptions.CHECKPOINTS_DIRECTORY, checkpointDir.toURI().toString());
+ config.set(
+ ExecutionCheckpointingOptions.EXTERNALIZED_CHECKPOINT,
+ CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
+ config.set(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 2);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(1), config, "firstPrefix");
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ final File completedCheckpoint = TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
+
+ config.set(SavepointConfigOptions.SAVEPOINT_PATH, completedCheckpoint.toURI().toString());
+
+ // Run a second job which aborts all lingering transactions and new consumer should
+ // immediately see the newly written records
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, "newPrefix");
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ @Test
+ public void testAbortTransactionsAfterScaleInBeforeFirstCheckpoint() throws Exception {
+ // Run a first job opening 5 transactions one per subtask and fail in async checkpoint phase
+ final Configuration config = new Configuration();
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 5);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(0), config, null);
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ assertTrue(deserializeValues(drainAllRecordsFromTopic(topic)).isEmpty());
+
+ // Second job aborts all transactions from previous runs with higher parallelism
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 1);
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, null);
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ private void executeWithMapper(
+ MapFunction<Long, Long> mapper,
+ Configuration config,
+ @Nullable String transactionalIdPrefix)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment(config);
+ env.enableCheckpointing(100L);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ final DataStreamSource<Long> source = env.fromSequence(1, 10);
+ final DataStream<Long> stream = source.map(mapper);
+ final KafkaSinkBuilder<Long> builder =
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic));
+ if (transactionalIdPrefix == null) {
+ transactionalIdPrefix = "kafka-sink";
+ }
+ builder.setTransactionalIdPrefix(transactionalIdPrefix);
+ stream.sinkTo(builder.build());
+ env.execute();
+ checkProducerLeak();
+ }
+
+ private void testRecoveryWithAssertion(
+ DeliveryGuarantee guarantee, java.util.function.Consumer<List<Long>> recordsAssertion)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment();
+ env.enableCheckpointing(300L);
+ DataStreamSource<Long> source = env.fromSequence(1, 10);
+ DataStream<Long> stream =
+ source.map(new FailingCheckpointMapper(failed, lastCheckpointedRecord));
+
+ stream.sinkTo(
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(guarantee)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic))
+ .setTransactionalIdPrefix("kafka-sink")
+ .build());
+ env.execute();
+
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ recordsAssertion.accept(deserializeValues(collectedRecords));
+ checkProducerLeak();
+ }
+
+ private void writeRecordsToKafka(
+ DeliveryGuarantee deliveryGuarantee, SharedReference<AtomicLong> expectedRecords)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment();
+ env.enableCheckpointing(1000L);
+ final DataStream<Long> source =
+ env.addSource(
+ new InfiniteIntegerSource(
+ emittedRecordsCount, emittedRecordsWithCheckpoint));
+ source.sinkTo(
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setDeliverGuarantee(deliveryGuarantee)
+ .setRecordSerializer(new RecordSerializer(topic))
+ .setTransactionalIdPrefix("kafka-sink")
+ .build());
+ env.execute();
+
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ final long recordsCount = expectedRecords.get().get();
+ assertEquals(collectedRecords.size(), recordsCount);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(0, recordsCount).boxed().collect(Collectors.toList()));
+ checkProducerLeak();
+ }
+
+ private static List<Long> deserializeValues(List<ConsumerRecord<byte[], byte[]>> records) {
+ return records.stream()
+ .map(
+ record -> {
+ final ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ final byte[] value = record.value();
+ buffer.put(value, 0, value.length);
+ buffer.flip();
+ return buffer.getLong();
+ })
+ .collect(Collectors.toList());
+ }
+
+ private static Properties getKafkaClientConfiguration() {
+ final Properties standardProps = new Properties();
+ standardProps.put("bootstrap.servers", KAFKA_CONTAINER.getBootstrapServers());
+ standardProps.put("group.id", UUID.randomUUID().toString());
+ standardProps.put("enable.auto.commit", false);
+ standardProps.put("auto.offset.reset", "earliest");
+ standardProps.put("max.partition.fetch.bytes", 256);
+ standardProps.put("zookeeper.session.timeout.ms", ZK_TIMEOUT_MILLIS);
+ standardProps.put("zookeeper.connection.timeout.ms", ZK_TIMEOUT_MILLIS);
+ return standardProps;
+ }
+
+ private Consumer<byte[], byte[]> createTestConsumer(String topic) {
+ final Properties consumerConfig = new Properties();
+ consumerConfig.putAll(getKafkaClientConfiguration());
+ consumerConfig.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ consumerConfig.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ consumerConfig.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ final KafkaConsumer<byte[], byte[]> kafkaConsumer = new KafkaConsumer<>(consumerConfig);
+ kafkaConsumer.subscribe(Collections.singletonList(topic));
+ return kafkaConsumer;
+ }
+
+ private void createTestTopic(String topic, int numPartitions, short replicationFactor)
+ throws ExecutionException, InterruptedException, TimeoutException {
+ Map<String, Object> properties = new HashMap<>();
+ properties.put(
+ CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
+ KAFKA_CONTAINER.getBootstrapServers());
+ try (AdminClient admin = AdminClient.create(properties)) {
+ final CreateTopicsResult result =
+ admin.createTopics(
+ Collections.singletonList(
+ new NewTopic(topic, numPartitions, replicationFactor)));
+ result.all().get(1, TimeUnit.MINUTES);
+ }
+ }
+
+ private void deleteTestTopic(String topic)
+ throws ExecutionException, InterruptedException, TimeoutException {
+ Map<String, Object> properties = new HashMap<>();
+ properties.put(
+ CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
+ KAFKA_CONTAINER.getBootstrapServers());
+ try (AdminClient admin = AdminClient.create(properties)) {
+ final DeleteTopicsResult result = admin.deleteTopics(Collections.singletonList(topic));
+ result.all().get(1, TimeUnit.MINUTES);
+ }
+ }
+
+ private List<ConsumerRecord<byte[], byte[]>> drainAllRecordsFromTopic(String topic) {
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords = new ArrayList<>();
+ try (Consumer<byte[], byte[]> consumer = createTestConsumer(topic)) {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ // Drain the kafka topic till all records are consumed
+ while (!records.isEmpty()) {
+ records.records(topic).forEach(collectedRecords::add);
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ }
+ return collectedRecords;
+ }
+
+ private static class RecordSerializer implements KafkaRecordSerializationSchema<Long> {
+
+ private final String topic;
+
+ public RecordSerializer(String topic) {
+ this.topic = topic;
+ }
+
+ @Override
+ public ProducerRecord<byte[], byte[]> serialize(
+ Long element, KafkaSinkContext context, Long timestamp) {
+ final ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ buffer.putLong(element);
+ return new ProducerRecord<>(topic, 0, null, null, buffer.array());
+ }
+ }
+
+ private static class FailAsyncCheckpointMapper
+ implements MapFunction<Long, Long>, CheckpointedFunction {
+ private static final ListStateDescriptor<Integer> stateDescriptor =
+ new ListStateDescriptor<>("test-state", new SlowSerializer());
+ private int failAfterCheckpoint;
+
+ private ListState<Integer> state;
+
+ public FailAsyncCheckpointMapper(int failAfterCheckpoint) {
+ this.failAfterCheckpoint = failAfterCheckpoint;
+ }
+
+ @Override
+ public Long map(Long value) throws Exception {
+ Thread.sleep(100);
+ return value;
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ state.clear();
+ if (failAfterCheckpoint <= 0) {
+ // Trigger a failure in the serializer
+ state.add(-1);
+ } else {
+ state.add(1);
+ }
+ failAfterCheckpoint--;
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ state = context.getOperatorStateStore().getListState(stateDescriptor);
+ }
+ }
+
+ private static class SlowSerializer extends TypeSerializerSingleton<Integer> {
+
+ @Override
+ public boolean isImmutableType() {
+ return false;
+ }
+
+ @Override
+ public Integer createInstance() {
+ return 1;
+ }
+
+ @Override
+ public Integer copy(Integer from) {
+ return from;
+ }
+
+ @Override
+ public Integer copy(Integer from, Integer reuse) {
+ return from;
+ }
+
+ @Override
+ public int getLength() {
+ return 0;
+ }
+
+ @Override
+ public void serialize(Integer record, DataOutputView target) throws IOException {
+ if (record != -1) {
+ return;
+ }
+ throw new RuntimeException("Expected failure during async checkpoint phase");
+ }
+
+ @Override
+ public Integer deserialize(DataInputView source) throws IOException {
+ return 1;
+ }
+
+ @Override
+ public Integer deserialize(Integer reuse, DataInputView source) throws IOException {
+ return 1;
+ }
+
+ @Override
+ public void copy(DataInputView source, DataOutputView target) throws IOException {}
+
+ @Override
+ public TypeSerializerSnapshot<Integer> snapshotConfiguration() {
+ return new SlowSerializerSnapshot();
+ }
+ }
+
+ /** Snapshot used in {@link FailAsyncCheckpointMapper}. */
+ public static class SlowSerializerSnapshot extends SimpleTypeSerializerSnapshot<Integer> {
+ public SlowSerializerSnapshot() {
+ super(SlowSerializer::new);
+ }
+ }
+
+ /** Fails after a checkpoint is taken and the next record was emitted. */
+ private static class FailingCheckpointMapper
+ implements MapFunction<Long, Long>, CheckpointListener, CheckpointedFunction {
+
+ private final SharedReference<AtomicBoolean> failed;
+ private final SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ private volatile long lastSeenRecord;
+ private volatile long checkpointedRecord;
+ private volatile long lastCheckpointId = 0;
+ private final AtomicInteger emittedBetweenCheckpoint = new AtomicInteger(0);
+
+ FailingCheckpointMapper(
+ SharedReference<AtomicBoolean> failed,
+ SharedReference<AtomicLong> lastCheckpointedRecord) {
+ this.failed = failed;
+ this.lastCheckpointedRecord = lastCheckpointedRecord;
+ }
+
+ @Override
+ public Long map(Long value) throws Exception {
+ lastSeenRecord = value;
+ if (lastCheckpointId >= 1
+ && emittedBetweenCheckpoint.get() > 0
+ && !failed.get().get()) {
+ failed.get().set(true);
+ throw new RuntimeException("Planned exception.");
+ }
+ Thread.sleep(50);
+ emittedBetweenCheckpoint.incrementAndGet();
+ return value;
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ lastCheckpointId = checkpointId;
+ emittedBetweenCheckpoint.set(0);
+ lastCheckpointedRecord.get().set(checkpointedRecord);
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ checkpointedRecord = lastSeenRecord;
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {}
+ }
+
+ private void checkProducerLeak() {
+ for (Thread t : Thread.getAllStackTraces().keySet()) {
+ if (t.getName().contains("kafka-producer-network-thread")) {
+ fail("Detected producer leak. Thread name: " + t.getName());
+ }
+ }
+ }
+
+ /**
+ * Exposes information about how man records have been emitted overall and at the beginning of a
+ * checkpoint.
+ */
+ private static final class InfiniteIntegerSource
Review comment:
Ah correct. Let's keep it this way then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r683151739
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/DefaultKafkaSinkContext.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Sink;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.common.PartitionInfo;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Context providing information to assist constructing a {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}.
+ */
+class DefaultKafkaSinkContext implements KafkaRecordSerializationSchema.KafkaSinkContext {
+
+ private final Sink.InitContext context;
+ private final Properties kafkaProducerConfig;
+
+ private int[] cachedPartitions;
+
+ public DefaultKafkaSinkContext(Sink.InitContext context, Properties kafkaProducerConfig) {
+ this.context = context;
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ }
+
+ @Override
+ public int getParallelInstanceId() {
+ return context.getSubtaskId();
+ }
+
+ @Override
+ public int getNumberOfParallelInstances() {
+ return context.getNumberOfParallelSubtasks();
+ }
+
+ @Override
Review comment:
```suggestion
@Override
public int[] getPartitionsForTopic(String topic) {
return cachedPartitions.computeIfAbsent(topic, this::fetchPartitions);
}
private int[] fetchPartitions(String topic) {
try (final Producer<?, ?> producer = new KafkaProducer<>(kafkaProducerConfig)) {
return producer.partitionsFor(topic).stream()
.mapToInt(PartitionInfo::partition)
.sorted()
.toArray();
}
}
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaCommittable.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+
+import java.util.Objects;
+
+/**
+ * This class holds the necessary information to construct a new {@link FlinkKafkaInternalProducer}
+ * to commit transactions in {@link KafkaCommitter}.
+ */
+class KafkaCommittable {
+
+ private final long producerId;
+ private final short epoch;
+ private final String transactionalId;
+
+ public KafkaCommittable(FlinkKafkaInternalProducer<byte[], byte[]> producer) {
+ this(
+ producer.getProducerId(),
+ producer.getEpoch(),
+ producer.getKafkaProducerConfig()
+ .getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG));
+ }
+
Review comment:
Nit: Please use a factory method when translating from one object to another.
See also https://www.programmersought.com/article/22326505572/
##########
File path: flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointTestBase.java
##########
@@ -165,35 +161,11 @@ protected File execute(UnalignedSettings settings) throws Exception {
miniCluster.after();
}
if (settings.generateCheckpoint) {
- return Files.find(checkpointDir.toPath(), 2, this::isCompletedCheckpoint)
- .max(Comparator.comparing(Path::toString))
- .map(Path::toFile)
- .orElseThrow(() -> new IllegalStateException("Cannot generate checkpoint"));
+ return TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
Review comment:
👍
Could you please also update CoordinatedSourceRescaleITCase?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} In this mode the KafkaSink will write all messages in
+ * a Kafka transaction that will be committed to Kafka on a checkpoint. It is very important to
+ * ensure that you use unique {@link #transactionalIdPrefix}s across your applications running
+ * on the same Kafka cluster otherwise I may happen that multiple running jobs corrupt their
+ * transactions!
Review comment:
```suggestion
* <li>{@link DeliveryGuarantee#EXACTLY_ONCE}: In this mode the KafkaSink will write all messages in
* a Kafka transaction that will be committed to Kafka on a checkpoint. Thus, if the consumer reads only committed data (see Kafka consumer config isolation.level), no duplicates will be seen in case of a Flink restart. However, this delays record writing effectively until a checkpoint is written, so adjust the checkpoint duration accordingly.
Please ensure that you use unique {@link #transactionalIdPrefix}s across your applications running
* on the same Kafka cluster such that multiple running jobs do not interfere in their
* transactions! Additionally, it is highly recommended to tweak Kafka transaction timeout (link) >> maximum checkpoint duration + maximum restart duration or data loss may happen when Kafka expires an uncommitted transaction.
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
+
+ private static byte readTransactionState(ByteBuffer buffer) {
+ // producerId
+ buffer.getLong();
+ // epoch
+ buffer.getShort();
+ // transactionTimeout
+ buffer.getInt();
+ // statusKey
+ return buffer.get();
+ }
Review comment:
```suggestion
private static byte readTransactionState(ByteBuffer buffer) {
return buffer.get(TRANSACTION_STATE_POS);
}
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
Review comment:
```suggestion
private void maybeAddTransaction(ConsumerRecord<byte[], byte[]> record,
Set<String> openTransactions) {
```
and do the lambda on call-site.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
+ recordSerializer.serialize(element, kafkaSinkContext, context.timestamp());
+ pendingRecords.incrementAndGet();
+ currentProducer.send(record, deliveryCallback);
+ }
+
+ @Override
+ public List<KafkaCommittable> prepareCommit(boolean flush) throws IOException {
+ flushRecords(flush);
+ if (!flush) {
+ currentProducer = beginTransaction();
+ }
+ final List<KafkaCommittable> committables = commit();
+ producers.add(currentProducer);
+ return committables;
+ }
+
+ @Override
+ public List<KafkaWriterState> snapshotState() throws IOException {
+ return ImmutableList.of(kafkaWriterState);
+ }
+
+ @Override
+ public void close() throws Exception {
+ currentProducer.close(Duration.ZERO);
+ }
+
+ private KafkaWriterState recoverAndInitializeState(List<KafkaWriterState> recoveredStates) {
+ final int subtaskId = kafkaSinkContext.getParallelInstanceId();
+ if (recoveredStates.isEmpty()) {
+ final KafkaWriterState state =
+ new KafkaWriterState(transactionalIdPrefix, subtaskId, 0);
+ abortTransactions(getTransactionsToAbort(state, new ArrayList<>()));
+ return state;
+ }
+ final Map<Integer, KafkaWriterState> taskOffsetMapping =
+ recoveredStates.stream()
+ .collect(
+ Collectors.toMap(
+ KafkaWriterState::getSubtaskId, Function.identity()));
+ if (!taskOffsetMapping.containsKey(subtaskId)) {
+ throw new IllegalStateException(
+ "It is expected that state from previous executions is distributed to the same subtask id.");
+ }
Review comment:
`checkState` also maybe add `internal error:`?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
+ * #setDeliverGuarantee(DeliveryGuarantee)} but keep in mind when using {@link
+ * DeliveryGuarantee#EXACTLY_ONCE} one must set the transactionalIdPrefix {@link
+ * #setTransactionalIdPrefix(String)}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+public class KafkaSinkBuilder<IN> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkBuilder.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private String transactionalIdPrefix = "kafka-sink";
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+ private String bootstrapServers;
+
+ /**
+ * Sets the wanted the {@link DeliveryGuarantee}. The default delivery guarantee is {@link
+ * #deliveryGuarantee}.
+ *
+ * @param deliveryGuarantee
+ * @return {@link KafkaSinkBuilder}
+ */
+ KafkaSinkBuilder<IN> setDeliverGuarantee(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "semantic");
+ return this;
+ }
+
+ /**
+ * Sets the configuration which used to instantiate all used {@link
+ * org.apache.kafka.clients.producer.KafkaProducer}.
+ *
+ * @param kafkaProducerConfig
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setKafkaProducerConfig(Properties kafkaProducerConfig) {
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ // set the producer configuration properties for kafka record key value serializers.
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG)) {
+ final long timeout = DEFAULT_KAFKA_TRANSACTION_TIMEOUT.toMillis();
+ checkState(
+ timeout < Integer.MAX_VALUE && timeout > 0,
+ "timeout does not fit into 32 bit integer");
+ kafkaProducerConfig.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, (int) timeout);
+ LOG.warn(
+ "Property [{}] not specified. Setting it to {}",
+ ProducerConfig.TRANSACTION_TIMEOUT_CONFIG,
+ DEFAULT_KAFKA_TRANSACTION_TIMEOUT);
+ }
+ return this;
+ }
+
+ /**
+ * Sets the {@link KafkaRecordSerializationSchema} that transforms incoming records to {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}s.
+ *
+ * @param recordSerializer
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setRecordSerializer(
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ ClosureCleaner.clean(
+ this.recordSerializer, ExecutionConfig.ClosureCleanerLevel.RECURSIVE, true);
+ return this;
+ }
+
+ /**
+ * Sets the prefix for all created transactionalIds if {@link DeliveryGuarantee#EXACTLY_ONCE} is
+ * configured.
+ *
+ * <p>It is mandatory to always set this value with {@link DeliveryGuarantee#EXACTLY_ONCE} to
+ * prevent corrupted transactions if multiple jobs using the KafkaSink run against the same
+ * Kafka Cluster. The default prefix is {@link #transactionalIdPrefix}.
+ *
+ * @param transactionalIdPrefix
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setTransactionalIdPrefix(String transactionalIdPrefix) {
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
Review comment:
Please add some length check. I think the maximum kafka transaction id length is 64k (please double check), so we could cap at 60k and still have some leeway.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
Review comment:
Just to be sure: this procedure would also abort all transactions from "unknown" subtasks because the checkpoint would not reflect the true previous parallelism (upscale-downscale without checkpoint), correct?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/DefaultKafkaSinkContext.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Sink;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.common.PartitionInfo;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Context providing information to assist constructing a {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}.
+ */
+class DefaultKafkaSinkContext implements KafkaRecordSerializationSchema.KafkaSinkContext {
+
+ private final Sink.InitContext context;
+ private final Properties kafkaProducerConfig;
+
+ private int[] cachedPartitions;
Review comment:
```suggestion
private final Map<String, int[]> cachedPartitions = new HashMap<>();
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/FlinkKafkaInternalProducer.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.TransactionalRequestResult;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Properties;
+
+/**
+ * A {@link KafkaProducer} that exposes private fields to allow resume producing from a given state.
+ */
+class FlinkKafkaInternalProducer<K, V> extends KafkaProducer<K, V> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkKafkaInternalProducer.class);
+ private static final String TRANSACTION_MANAGER_STATE_ENUM =
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State";
+
+ private final Properties kafkaProducerConfig;
+ @Nullable private final String transactionalId;
+
+ public FlinkKafkaInternalProducer(Properties properties) {
+ super(properties);
+ this.kafkaProducerConfig = properties;
+ this.transactionalId = properties.getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ if (transactionalId != null) {
+ flushNewPartitions();
+ }
+ }
+
+ public Properties getKafkaProducerConfig() {
+ return kafkaProducerConfig;
+ }
+
+ public short getEpoch() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (short) getField(producerIdAndEpoch, "epoch");
+ }
+
+ public long getProducerId() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (long) getField(producerIdAndEpoch, "producerId");
+ }
+
+ /**
+ * Besides committing {@link org.apache.kafka.clients.producer.KafkaProducer#commitTransaction}
+ * is also adding new partitions to the transaction. flushNewPartitions method is moving this
+ * logic to pre-commit/flush, to make resumeTransaction simpler. Otherwise resumeTransaction
+ * would require to restore state of the not yet added/"in-flight" partitions.
+ */
+ private void flushNewPartitions() {
+ LOG.info("Flushing new partitions");
+ TransactionalRequestResult result = enqueueNewPartitions();
+ Object sender = getField("sender");
+ invoke(sender, "wakeup");
+ result.await();
+ }
+
+ /**
+ * Enqueues new transactions at the transaction manager and returns a {@link
+ * TransactionalRequestResult} that allows waiting on them.
+ *
+ * <p>If there are no new transactions we return a {@link TransactionalRequestResult} that is
+ * already done.
+ */
+ private TransactionalRequestResult enqueueNewPartitions() {
+ Object transactionManager = getField("transactionManager");
+ synchronized (transactionManager) {
+ Object newPartitionsInTransaction =
+ getField(transactionManager, "newPartitionsInTransaction");
+ Object newPartitionsInTransactionIsEmpty =
+ invoke(newPartitionsInTransaction, "isEmpty");
+ TransactionalRequestResult result;
+ if (newPartitionsInTransactionIsEmpty instanceof Boolean
+ && !((Boolean) newPartitionsInTransactionIsEmpty)) {
+ Object txnRequestHandler =
+ invoke(transactionManager, "addPartitionsToTransactionHandler");
+ invoke(
+ transactionManager,
+ "enqueueRequest",
+ new Class[] {txnRequestHandler.getClass().getSuperclass()},
+ new Object[] {txnRequestHandler});
+ result =
+ (TransactionalRequestResult)
+ getField(
+ txnRequestHandler,
+ txnRequestHandler.getClass().getSuperclass(),
+ "result");
+ } else {
+ // we don't have an operation but this operation string is also used in
+ // addPartitionsToTransactionHandler.
+ result = new TransactionalRequestResult("AddPartitionsToTxn");
+ result.done();
+ }
+ return result;
+ }
+ }
+
+ private static Object invoke(Object object, String methodName, Object... args) {
+ Class<?>[] argTypes = new Class[args.length];
+ for (int i = 0; i < args.length; i++) {
+ argTypes[i] = args[i].getClass();
+ }
+ return invoke(object, methodName, argTypes, args);
+ }
+
+ private static Object invoke(
+ Object object, String methodName, Class<?>[] argTypes, Object[] args) {
+ try {
+ Method method = object.getClass().getDeclaredMethod(methodName, argTypes);
+ method.setAccessible(true);
+ return method.invoke(object, args);
+ } catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
Review comment:
We should switch to Guava's Invokable here.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaCommittableSerializer.java
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+
+class KafkaCommittableSerializer implements SimpleVersionedSerializer<KafkaCommittable> {
+
+ @Override
+ public int getVersion() {
+ return 1;
+ }
+
+ @Override
+ public byte[] serialize(KafkaCommittable state) throws IOException {
+ try (final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ final DataOutputStream out = new DataOutputStream(baos)) {
+ out.writeShort(state.getEpoch());
+ out.writeLong(state.getProducerId());
+ out.writeUTF(state.getTransactionalId());
Review comment:
Just FYI: `writeUTF` is limited to 64k strings, so always be alert when you see it. However, Kafka transactions have the same limit so this is fine.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
Review comment:
double negation ;)
```suggestion
* <li>{@link DeliveryGuarantee#NONE} does not provide any guarantees: messages may be lost in case of issues on the Kafka broker and messages may be duplicated in case of a Flink failure.
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaCommitter.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.errors.InvalidTxnStateException;
+import org.apache.kafka.common.errors.ProducerFencedException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Committer implementation for {@link KafkaSink}
+ *
+ * <p>The committer is responsible to finalize the Kafka transactions by committing them.
+ */
+class KafkaCommitter implements Committer<KafkaCommittable> {
+
+ private final Properties kafkaProducerConfig;
+
+ KafkaCommitter(Properties kafkaProducerConfig) {
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ }
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaCommitter.class);
+
+ @Override
+ public List<KafkaCommittable> commit(List<KafkaCommittable> committables) throws IOException {
+ committables.forEach(this::commitTransaction);
+ return Collections.emptyList();
+ }
+
+ @Override
+ public void close() throws Exception {}
+
+ private void commitTransaction(KafkaCommittable committable) {
+ final String transactionalId = committable.getTransactionalId();
+ LOG.debug("Committing Kafka transaction {}", transactionalId);
+ try (FlinkKafkaInternalProducer<?, ?> producer =
+ new FlinkKafkaInternalProducer<>(createKafkaProducerConfig(transactionalId))) {
+ producer.resumeTransaction(committable.getProducerId(), committable.getEpoch());
+ producer.commitTransaction();
+ } catch (InvalidTxnStateException | ProducerFencedException e) {
+ // That means we have committed this transaction before.
+ LOG.warn(
+ "Encountered error {} while recovering transaction {}. "
+ + "Presumably this transaction has been already committed before",
+ e,
+ committable);
Review comment:
I'm wondering if we need to do something else, see first mail in `Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure` on user ML.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/FlinkKafkaInternalProducer.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.TransactionalRequestResult;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Properties;
+
+/**
+ * A {@link KafkaProducer} that exposes private fields to allow resume producing from a given state.
+ */
+class FlinkKafkaInternalProducer<K, V> extends KafkaProducer<K, V> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkKafkaInternalProducer.class);
+ private static final String TRANSACTION_MANAGER_STATE_ENUM =
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State";
+
+ private final Properties kafkaProducerConfig;
+ @Nullable private final String transactionalId;
+
+ public FlinkKafkaInternalProducer(Properties properties) {
+ super(properties);
+ this.kafkaProducerConfig = properties;
+ this.transactionalId = properties.getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ if (transactionalId != null) {
+ flushNewPartitions();
+ }
+ }
+
+ public Properties getKafkaProducerConfig() {
+ return kafkaProducerConfig;
+ }
+
+ public short getEpoch() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
Review comment:
Extract constants for often used fields (lets say >2 usages).
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
Review comment:
```suggestion
* <li>{@link DeliveryGuarantee#AT_LEAST_ONCE}: the sink will wait for all outstanding records in the
* Kafka buffers to be acknowledged by the Kafka producer on a checkpoint. No messages will be lost in case of any issue with the Kafka brokers but messages may be duplicated when Flink restarts.
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/FlinkKafkaInternalProducer.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.TransactionalRequestResult;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Properties;
+
+/**
+ * A {@link KafkaProducer} that exposes private fields to allow resume producing from a given state.
+ */
+class FlinkKafkaInternalProducer<K, V> extends KafkaProducer<K, V> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkKafkaInternalProducer.class);
+ private static final String TRANSACTION_MANAGER_STATE_ENUM =
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State";
+
+ private final Properties kafkaProducerConfig;
+ @Nullable private final String transactionalId;
+
+ public FlinkKafkaInternalProducer(Properties properties) {
+ super(properties);
+ this.kafkaProducerConfig = properties;
+ this.transactionalId = properties.getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ if (transactionalId != null) {
+ flushNewPartitions();
+ }
+ }
+
+ public Properties getKafkaProducerConfig() {
+ return kafkaProducerConfig;
+ }
+
+ public short getEpoch() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (short) getField(producerIdAndEpoch, "epoch");
+ }
+
+ public long getProducerId() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (long) getField(producerIdAndEpoch, "producerId");
+ }
+
+ /**
+ * Besides committing {@link org.apache.kafka.clients.producer.KafkaProducer#commitTransaction}
+ * is also adding new partitions to the transaction. flushNewPartitions method is moving this
+ * logic to pre-commit/flush, to make resumeTransaction simpler. Otherwise resumeTransaction
+ * would require to restore state of the not yet added/"in-flight" partitions.
+ */
+ private void flushNewPartitions() {
+ LOG.info("Flushing new partitions");
+ TransactionalRequestResult result = enqueueNewPartitions();
+ Object sender = getField("sender");
+ invoke(sender, "wakeup");
+ result.await();
+ }
+
+ /**
+ * Enqueues new transactions at the transaction manager and returns a {@link
+ * TransactionalRequestResult} that allows waiting on them.
+ *
+ * <p>If there are no new transactions we return a {@link TransactionalRequestResult} that is
+ * already done.
+ */
+ private TransactionalRequestResult enqueueNewPartitions() {
+ Object transactionManager = getField("transactionManager");
+ synchronized (transactionManager) {
+ Object newPartitionsInTransaction =
+ getField(transactionManager, "newPartitionsInTransaction");
+ Object newPartitionsInTransactionIsEmpty =
+ invoke(newPartitionsInTransaction, "isEmpty");
+ TransactionalRequestResult result;
+ if (newPartitionsInTransactionIsEmpty instanceof Boolean
+ && !((Boolean) newPartitionsInTransactionIsEmpty)) {
+ Object txnRequestHandler =
+ invoke(transactionManager, "addPartitionsToTransactionHandler");
+ invoke(
+ transactionManager,
+ "enqueueRequest",
+ new Class[] {txnRequestHandler.getClass().getSuperclass()},
+ new Object[] {txnRequestHandler});
+ result =
+ (TransactionalRequestResult)
+ getField(
+ txnRequestHandler,
+ txnRequestHandler.getClass().getSuperclass(),
+ "result");
+ } else {
+ // we don't have an operation but this operation string is also used in
+ // addPartitionsToTransactionHandler.
+ result = new TransactionalRequestResult("AddPartitionsToTxn");
+ result.done();
+ }
+ return result;
+ }
+ }
+
+ private static Object invoke(Object object, String methodName, Object... args) {
+ Class<?>[] argTypes = new Class[args.length];
+ for (int i = 0; i < args.length; i++) {
+ argTypes[i] = args[i].getClass();
+ }
+ return invoke(object, methodName, argTypes, args);
+ }
+
+ private static Object invoke(
+ Object object, String methodName, Class<?>[] argTypes, Object[] args) {
+ try {
+ Method method = object.getClass().getDeclaredMethod(methodName, argTypes);
+ method.setAccessible(true);
+ return method.invoke(object, args);
+ } catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ private Object getField(String fieldName) {
+ return getField(this, KafkaProducer.class, fieldName);
+ }
+
+ /**
+ * Gets and returns the field {@code fieldName} from the given Object {@code object} using
+ * reflection.
+ */
+ private static Object getField(Object object, String fieldName) {
+ return getField(object, object.getClass(), fieldName);
+ }
+
+ /**
+ * Gets and returns the field {@code fieldName} from the given Object {@code object} using
+ * reflection.
+ */
+ private static Object getField(Object object, Class<?> clazz, String fieldName) {
+ try {
+ Field field = clazz.getDeclaredField(fieldName);
+ field.setAccessible(true);
+ return field.get(object);
+ } catch (NoSuchFieldException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ /**
+ * Instead of obtaining producerId and epoch from the transaction coordinator, re-use previously
+ * obtained ones, so that we can resume transaction after a restart. Implementation of this
+ * method is based on {@link KafkaProducer#initTransactions}.
+ * https://github.com/apache/kafka/commit/5d2422258cb975a137a42a4e08f03573c49a387e#diff-f4ef1afd8792cd2a2e9069cd7ddea630
+ */
+ public void resumeTransaction(long producerId, short epoch) {
+ Preconditions.checkState(
+ producerId >= 0 && epoch >= 0,
+ "Incorrect values for producerId %s and epoch %s",
+ producerId,
+ epoch);
+ LOG.info(
+ "Attempting to resume transaction {} with producerId {} and epoch {}",
+ transactionalId,
+ producerId,
+ epoch);
+
+ Object transactionManager = getField("transactionManager");
+ synchronized (transactionManager) {
+ Object topicPartitionBookkeeper =
+ getField(transactionManager, "topicPartitionBookkeeper");
+
+ invoke(
+ transactionManager,
+ "transitionTo",
+ createTransactionManagerState("INITIALIZING"));
+ invoke(topicPartitionBookkeeper, "reset");
+
+ setField(
+ transactionManager,
+ "producerIdAndEpoch",
+ createProducerIdAndEpoch(producerId, epoch));
+
+ invoke(transactionManager, "transitionTo", createTransactionManagerState("READY"));
+
+ invoke(
+ transactionManager,
+ "transitionTo",
+ createTransactionManagerState("IN_TRANSACTION"));
+ setField(transactionManager, "transactionStarted", true);
+ }
+ }
+
+ private static Object createProducerIdAndEpoch(long producerId, short epoch) {
+ try {
+ Class<?> clazz =
+ Class.forName("org.apache.kafka.clients.producer.internals.ProducerIdAndEpoch");
+ Constructor<?> constructor = clazz.getDeclaredConstructors()[0];
+ constructor.setAccessible(true);
+ return constructor.newInstance(producerId, epoch);
+ } catch (ClassNotFoundException
+ | InvocationTargetException
+ | InstantiationException
+ | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ /**
+ * Sets the field {@code fieldName} on the given Object {@code object} to {@code value} using
+ * reflection.
+ */
+ private static void setField(Object object, String fieldName, Object value) {
+ try {
+ Field field = object.getClass().getDeclaredField(fieldName);
+ field.setAccessible(true);
+ field.set(object, value);
+ } catch (NoSuchFieldException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ private static Enum<?> createTransactionManagerState(String enumName) {
Review comment:
```suggestion
private static Enum<?> getTransactionManagerState(String enumName) {
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} In this mode the KafkaSink will write all messages in
+ * a Kafka transaction that will be committed to Kafka on a checkpoint. It is very important to
+ * ensure that you use unique {@link #transactionalIdPrefix}s across your applications running
+ * on the same Kafka cluster otherwise I may happen that multiple running jobs corrupt their
+ * transactions!
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
Review comment:
I'd skip it here. That's what the builder is for.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} In this mode the KafkaSink will write all messages in
+ * a Kafka transaction that will be committed to Kafka on a checkpoint. It is very important to
+ * ensure that you use unique {@link #transactionalIdPrefix}s across your applications running
+ * on the same Kafka cluster otherwise I may happen that multiple running jobs corrupt their
+ * transactions!
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
+ *
+ * @param <IN> type of the records written to Kafka
+ * @see KafkaSinkBuilder on how construct a KafkaSink
Review comment:
```suggestion
* @see KafkaSinkBuilder on how to construct a KafkaSink
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
+ * #setDeliverGuarantee(DeliveryGuarantee)} but keep in mind when using {@link
+ * DeliveryGuarantee#EXACTLY_ONCE} one must set the transactionalIdPrefix {@link
+ * #setTransactionalIdPrefix(String)}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+public class KafkaSinkBuilder<IN> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkBuilder.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private String transactionalIdPrefix = "kafka-sink";
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+ private String bootstrapServers;
+
+ /**
+ * Sets the wanted the {@link DeliveryGuarantee}. The default delivery guarantee is {@link
+ * #deliveryGuarantee}.
+ *
+ * @param deliveryGuarantee
+ * @return {@link KafkaSinkBuilder}
+ */
+ KafkaSinkBuilder<IN> setDeliverGuarantee(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "semantic");
+ return this;
+ }
+
+ /**
+ * Sets the configuration which used to instantiate all used {@link
+ * org.apache.kafka.clients.producer.KafkaProducer}.
+ *
+ * @param kafkaProducerConfig
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setKafkaProducerConfig(Properties kafkaProducerConfig) {
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ // set the producer configuration properties for kafka record key value serializers.
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG)) {
+ final long timeout = DEFAULT_KAFKA_TRANSACTION_TIMEOUT.toMillis();
+ checkState(
+ timeout < Integer.MAX_VALUE && timeout > 0,
+ "timeout does not fit into 32 bit integer");
+ kafkaProducerConfig.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, (int) timeout);
+ LOG.warn(
+ "Property [{}] not specified. Setting it to {}",
+ ProducerConfig.TRANSACTION_TIMEOUT_CONFIG,
+ DEFAULT_KAFKA_TRANSACTION_TIMEOUT);
+ }
+ return this;
+ }
+
+ /**
+ * Sets the {@link KafkaRecordSerializationSchema} that transforms incoming records to {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}s.
+ *
+ * @param recordSerializer
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setRecordSerializer(
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ ClosureCleaner.clean(
+ this.recordSerializer, ExecutionConfig.ClosureCleanerLevel.RECURSIVE, true);
+ return this;
+ }
+
+ /**
+ * Sets the prefix for all created transactionalIds if {@link DeliveryGuarantee#EXACTLY_ONCE} is
+ * configured.
+ *
+ * <p>It is mandatory to always set this value with {@link DeliveryGuarantee#EXACTLY_ONCE} to
+ * prevent corrupted transactions if multiple jobs using the KafkaSink run against the same
+ * Kafka Cluster. The default prefix is {@link #transactionalIdPrefix}.
+ *
+ * @param transactionalIdPrefix
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setTransactionalIdPrefix(String transactionalIdPrefix) {
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ return this;
+ }
+
+ /**
+ * Sets the Kafka bootstrap servers.
+ *
+ * @param bootstrapServers a comma separated list of valid URIs to reach the Kafka broker
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setBootstrapServers(String bootstrapServers) {
+ this.bootstrapServers = checkNotNull(bootstrapServers);
+ return this;
+ }
+
+ /**
+ * Constructs the {@link KafkaSink} with the configured properties.
+ *
+ * @return {@link KafkaSink}
+ */
+ public KafkaSink<IN> build() {
+ checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ checkNotNull(bootstrapServers);
+ if (deliveryGuarantee == DeliveryGuarantee.EXACTLY_ONCE) {
+ checkNotNull(
+ transactionalIdPrefix,
+ "EXACTLY_ONCE delivery guarantee requires a transactionIdPrefix to be set to provide unique transaction names across multiple KafkaSinks writing to the same Kafka cluster.");
+ }
Review comment:
nit: `checkNotNull` is typically used for parameters here `checkState` would be more appropriate. (KafkaSource is doing the same mistake)
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
Review comment:
Add a link to the KafkaSink description of guarantees.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
Review comment:
How about passing the combined list as parameter and then extract main as the first element?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
+ * #setDeliverGuarantee(DeliveryGuarantee)} but keep in mind when using {@link
+ * DeliveryGuarantee#EXACTLY_ONCE} one must set the transactionalIdPrefix {@link
+ * #setTransactionalIdPrefix(String)}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+public class KafkaSinkBuilder<IN> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkBuilder.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private String transactionalIdPrefix = "kafka-sink";
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+ private String bootstrapServers;
+
+ /**
+ * Sets the wanted the {@link DeliveryGuarantee}. The default delivery guarantee is {@link
+ * #deliveryGuarantee}.
+ *
+ * @param deliveryGuarantee
+ * @return {@link KafkaSinkBuilder}
+ */
+ KafkaSinkBuilder<IN> setDeliverGuarantee(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "semantic");
Review comment:
```suggestion
this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
```
or leave that param out as discussed.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
Review comment:
id seems to be UTF-8 encoded
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
Review comment:
no copy :p
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
+
+ private static byte readTransactionState(ByteBuffer buffer) {
+ // producerId
+ buffer.getLong();
+ // epoch
+ buffer.getShort();
+ // transactionTimeout
+ buffer.getInt();
+ // statusKey
+ return buffer.get();
+ }
+
+ enum TransactionState {
+ Empty(Byte.parseByte("0")),
+ Ongoing(Byte.parseByte("1")),
+ PrepareCommit(Byte.parseByte("2")),
+ PrepareAbort(Byte.parseByte("3")),
+ CompleteCommit(Byte.parseByte("4")),
+ CompleteAbort(Byte.parseByte("5")),
+ Dead(Byte.parseByte("6")),
+ PrepareEpochFence(Byte.parseByte("7"));
+
+ private final byte state;
+
+ TransactionState(byte state) {
+ this.state = state;
+ }
+
+ static TransactionState fromByte(byte state) {
+ return Arrays.stream(TransactionState.values())
+ .filter((value) -> value.state == state)
+ .findFirst()
+ .orElseThrow(
+ () ->
+ new IllegalArgumentException(
+ String.format(
+ "The given state %s is not supported.",
+ state)));
+ }
Review comment:
```suggestion
private static final Map<Byte, TransactionState> BYTE_TO_STATE =
Arrays.stream(TransactionState.values())
.collect(Collectors.toMap(e -> e.state, e -> e));
static TransactionState fromByte(byte state) {
TransactionState transactionState = BYTE_TO_STATE.get(state);
if (transactionState == null) {
throw new IllegalArgumentException(
String.format("The given state %s is not supported.", state));
}
return transactionState;
}
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
+
+ private static byte readTransactionState(ByteBuffer buffer) {
+ // producerId
+ buffer.getLong();
+ // epoch
+ buffer.getShort();
+ // transactionTimeout
+ buffer.getInt();
+ // statusKey
+ return buffer.get();
+ }
+
+ enum TransactionState {
+ Empty(Byte.parseByte("0")),
Review comment:
Welcome to Java! `Empty((byte) 0)`
Alternatively just have the ctor
```
TransactionState(int state) {
this.state = (byte) state;
}
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
Review comment:
Parsing should happen after prefix check.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
Review comment:
This is quite whacky:
- It may read too few records if Kafka broker has a small hiccup (1s)
- It most certainly reads too much if other applications append new transactions ever 1s (I'm assuming that this is not unusual for larger setups)
You should fetch the current committed offset before entering the loop and only break the loop iff the offset is reached.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
Review comment:
You should have a list of states in case of downscale (first entry should be the `main` or `active` entry).
Currently, we abort all transactions that are beyond this state, so if a restart happens before we committed everything, we may actually abort the transaction because we only store the `main` state in a new checkpoint. However, that would also not work with our enumeration of unknown task approach.
We probably need to encode the attempt after all ;) Can you create a follow-up ticket? The current limitation is that we assume that all transactions can be committed in-between downscale and the next checkpoint, which is okayish for a start.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/DefaultKafkaSinkContext.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Sink;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.common.PartitionInfo;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Context providing information to assist constructing a {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}.
+ */
+class DefaultKafkaSinkContext implements KafkaRecordSerializationSchema.KafkaSinkContext {
+
+ private final Sink.InitContext context;
Review comment:
Using the `InitContext` here makes it hard to create for testing. I'd pass the parallelism things directly.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
Review comment:
Please extract that logic into a test method and add unit tests + comments.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
+ recordSerializer.serialize(element, kafkaSinkContext, context.timestamp());
+ pendingRecords.incrementAndGet();
+ currentProducer.send(record, deliveryCallback);
+ }
+
+ @Override
+ public List<KafkaCommittable> prepareCommit(boolean flush) throws IOException {
+ flushRecords(flush);
+ if (!flush) {
+ currentProducer = beginTransaction();
+ }
+ final List<KafkaCommittable> committables = commit();
+ producers.add(currentProducer);
+ return committables;
+ }
+
+ @Override
+ public List<KafkaWriterState> snapshotState() throws IOException {
+ return ImmutableList.of(kafkaWriterState);
+ }
+
+ @Override
+ public void close() throws Exception {
+ currentProducer.close(Duration.ZERO);
Review comment:
producers for each?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
Review comment:
At this point we should immediately filter if the prefix matches.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
Review comment:
We should avoid `record` ;)
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
Review comment:
We should now enqueue a mail to avoid the `AtomicLong` and `volatile`.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
Review comment:
I'd index combined by subtask id for a non-scanning lookup.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
Review comment:
I'd also add a link in the first sentence such that it appears in all javadocs.
##########
File path: flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/DeliveryGuarantee.java
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base;
+
+/**
+ * Semantics that can be chosen.
+ * <li>{@link #EXACTLY_ONCE}
+ * <li>{@link #AT_LEAST_ONCE}
+ * <li>{@link #NONE}
+ */
+public enum DeliveryGuarantee {
+ EXACTLY_ONCE,
Review comment:
Thanks for expanding the javadoc. I was wondering if it wouldn't be better to move the javadoc for the individual constants to the constants? ;)
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriterState.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class KafkaWriterState {
+ private final int subtaskId;
+ private final long transactionalIdOffset;
+ private final String transactionalIdPrefix;
Review comment:
By storing it here, do you effectively allow users to change the prefix even when resuming from checkpoint?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
+ recordSerializer.serialize(element, kafkaSinkContext, context.timestamp());
+ pendingRecords.incrementAndGet();
+ currentProducer.send(record, deliveryCallback);
+ }
+
+ @Override
+ public List<KafkaCommittable> prepareCommit(boolean flush) throws IOException {
+ flushRecords(flush);
+ if (!flush) {
+ currentProducer = beginTransaction();
+ }
+ final List<KafkaCommittable> committables = commit();
+ producers.add(currentProducer);
+ return committables;
+ }
+
+ @Override
+ public List<KafkaWriterState> snapshotState() throws IOException {
+ return ImmutableList.of(kafkaWriterState);
+ }
+
+ @Override
+ public void close() throws Exception {
+ currentProducer.close(Duration.ZERO);
+ }
+
+ private KafkaWriterState recoverAndInitializeState(List<KafkaWriterState> recoveredStates) {
+ final int subtaskId = kafkaSinkContext.getParallelInstanceId();
+ if (recoveredStates.isEmpty()) {
+ final KafkaWriterState state =
+ new KafkaWriterState(transactionalIdPrefix, subtaskId, 0);
+ abortTransactions(getTransactionsToAbort(state, new ArrayList<>()));
+ return state;
+ }
+ final Map<Integer, KafkaWriterState> taskOffsetMapping =
+ recoveredStates.stream()
+ .collect(
+ Collectors.toMap(
+ KafkaWriterState::getSubtaskId, Function.identity()));
+ if (!taskOffsetMapping.containsKey(subtaskId)) {
+ throw new IllegalStateException(
+ "It is expected that state from previous executions is distributed to the same subtask id.");
+ }
+ final KafkaWriterState lastState = taskOffsetMapping.get(subtaskId);
+ taskOffsetMapping.remove(subtaskId);
+ abortTransactions(
+ getTransactionsToAbort(lastState, new ArrayList<>(taskOffsetMapping.values())));
+ if (!lastState.getTransactionalIdPrefix().equals(transactionalIdPrefix)) {
+ LOG.warn(
+ "Transactional id prefix from previous execution {} has changed to {}.",
+ lastState.getTransactionalIdPrefix(),
+ transactionalIdPrefix);
+ return new KafkaWriterState(transactionalIdPrefix, subtaskId, 0);
+ }
+ return new KafkaWriterState(
+ transactionalIdPrefix, subtaskId, lastState.getTransactionalIdOffset());
+ }
+
+ private void abortTransactions(Set<String> transactionsToAbort) {
+ transactionsToAbort.forEach(
+ transaction -> {
+ // don't mess with the original configuration or any other
+ // properties of the
+ // original object
+ // -> create an internal kafka producer on our own and do not rely
+ // on
+ // initTransactionalProducer().
+ final Properties myConfig = new Properties();
+ myConfig.putAll(kafkaProducerConfig);
+ myConfig.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transaction);
+ LOG.info("Aborting Kafka transaction {}.", transaction);
+ FlinkKafkaInternalProducer<byte[], byte[]> kafkaProducer = null;
Review comment:
We could use try-with-resource if we relay `close` to `close(0)` in `FlinkKafkaInternalProducer`.
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkITCase.java
##########
@@ -0,0 +1,634 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.StateBackendOptions;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.test.util.TestUtils;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.testutils.junit.SharedReference;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.CreateTopicsResult;
+import org.apache.kafka.clients.admin.DeleteTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.LongStream;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+/** Tests for using KafkaSink writing to a Kafka cluster. */
+public class KafkaSinkITCase extends TestLogger {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkITCase.class);
+ private static final Slf4jLogConsumer LOG_CONSUMER = new Slf4jLogConsumer(LOG);
+ private static final String INTER_CONTAINER_KAFKA_ALIAS = "kafka";
+ private static final Network NETWORK = Network.newNetwork();
+ private static final int ZK_TIMEOUT_MILLIS = 30000;
+ private static final short TOPIC_REPLICATION_FACTOR = 1;
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+
+ private String topic;
+ private SharedReference<AtomicLong> emittedRecordsCount;
+ private SharedReference<AtomicLong> emittedRecordsWithCheckpoint;
+ private SharedReference<AtomicBoolean> failed;
+ private SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ @ClassRule
+ public static final KafkaContainer KAFKA_CONTAINER =
+ new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.5.2"))
+ .withEmbeddedZookeeper()
+ .withEnv(
+ ImmutableMap.of(
+ "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR",
+ "1",
+ "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
+ String.valueOf(Duration.ofHours(2).toMillis()),
+ "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR",
+ "1",
+ "KAFKA_MIN_INSYNC_REPLICAS",
+ "1"))
+ .withNetwork(NETWORK)
+ .withLogConsumer(LOG_CONSUMER)
+ .withNetworkAliases(INTER_CONTAINER_KAFKA_ALIAS);
+
+ @Rule public final SharedObjects sharedObjects = SharedObjects.create();
+
+ @Rule public final TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void setUp() throws ExecutionException, InterruptedException, TimeoutException {
+ emittedRecordsCount = sharedObjects.add(new AtomicLong());
+ emittedRecordsWithCheckpoint = sharedObjects.add(new AtomicLong());
+ failed = sharedObjects.add(new AtomicBoolean(false));
+ lastCheckpointedRecord = sharedObjects.add(new AtomicLong(0));
+ topic = UUID.randomUUID().toString();
+ createTestTopic(topic, 1, TOPIC_REPLICATION_FACTOR);
+ }
+
+ @After
+ public void tearDown() throws ExecutionException, InterruptedException, TimeoutException {
+ deleteTestTopic(topic);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithAtLeastOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.AT_LEAST_ONCE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithNoneGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.NONE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithExactlyOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.EXACTLY_ONCE, emittedRecordsWithCheckpoint);
+ }
+
+ @Test
+ public void testRecoveryWithAtLeastOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.AT_LEAST_ONCE,
+ (records) ->
+ assertThat(records, hasItems(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L)));
+ }
+
+ @Test
+ public void testRecoveryWithExactlyOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.EXACTLY_ONCE,
+ (records) ->
+ assertEquals(
+ records,
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList())));
+ }
+
+ @Test
+ public void testAbortTransactionsOfPendingCheckpointsAfterFailure() throws Exception {
+ // Run a first job failing during the async phase of a checkpoint to leave some
+ // lingering transactions
+ final Configuration config = new Configuration();
+ config.setString(StateBackendOptions.STATE_BACKEND, "filesystem");
+ final File checkpointDir = temp.newFolder();
+ config.setString(
+ CheckpointingOptions.CHECKPOINTS_DIRECTORY, checkpointDir.toURI().toString());
+ config.set(
+ ExecutionCheckpointingOptions.EXTERNALIZED_CHECKPOINT,
+ CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
+ config.set(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 2);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(1), config, "firstPrefix");
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ final File completedCheckpoint = TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
+
+ config.set(SavepointConfigOptions.SAVEPOINT_PATH, completedCheckpoint.toURI().toString());
+
+ // Run a second job which aborts all lingering transactions and new consumer should
+ // immediately see the newly written records
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, "newPrefix");
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ @Test
+ public void testAbortTransactionsAfterScaleInBeforeFirstCheckpoint() throws Exception {
+ // Run a first job opening 5 transactions one per subtask and fail in async checkpoint phase
+ final Configuration config = new Configuration();
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 5);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(0), config, null);
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ assertTrue(deserializeValues(drainAllRecordsFromTopic(topic)).isEmpty());
+
+ // Second job aborts all transactions from previous runs with higher parallelism
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 1);
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, null);
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ private void executeWithMapper(
+ MapFunction<Long, Long> mapper,
+ Configuration config,
+ @Nullable String transactionalIdPrefix)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment(config);
+ env.enableCheckpointing(100L);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ final DataStreamSource<Long> source = env.fromSequence(1, 10);
+ final DataStream<Long> stream = source.map(mapper);
+ final KafkaSinkBuilder<Long> builder =
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic));
+ if (transactionalIdPrefix == null) {
+ transactionalIdPrefix = "kafka-sink";
+ }
+ builder.setTransactionalIdPrefix(transactionalIdPrefix);
+ stream.sinkTo(builder.build());
+ env.execute();
+ checkProducerLeak();
+ }
+
+ private void testRecoveryWithAssertion(
+ DeliveryGuarantee guarantee, java.util.function.Consumer<List<Long>> recordsAssertion)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment();
+ env.enableCheckpointing(300L);
+ DataStreamSource<Long> source = env.fromSequence(1, 10);
+ DataStream<Long> stream =
+ source.map(new FailingCheckpointMapper(failed, lastCheckpointedRecord));
+
+ stream.sinkTo(
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(guarantee)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic))
+ .setTransactionalIdPrefix("kafka-sink")
+ .build());
+ env.execute();
+
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ recordsAssertion.accept(deserializeValues(collectedRecords));
+ checkProducerLeak();
+ }
+
+ private void writeRecordsToKafka(
+ DeliveryGuarantee deliveryGuarantee, SharedReference<AtomicLong> expectedRecords)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment();
+ env.enableCheckpointing(1000L);
+ final DataStream<Long> source =
+ env.addSource(
+ new InfiniteIntegerSource(
+ emittedRecordsCount, emittedRecordsWithCheckpoint));
+ source.sinkTo(
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setDeliverGuarantee(deliveryGuarantee)
+ .setRecordSerializer(new RecordSerializer(topic))
+ .setTransactionalIdPrefix("kafka-sink")
+ .build());
+ env.execute();
+
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ final long recordsCount = expectedRecords.get().get();
+ assertEquals(collectedRecords.size(), recordsCount);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(0, recordsCount).boxed().collect(Collectors.toList()));
+ checkProducerLeak();
+ }
+
+ private static List<Long> deserializeValues(List<ConsumerRecord<byte[], byte[]>> records) {
+ return records.stream()
+ .map(
+ record -> {
+ final ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ final byte[] value = record.value();
+ buffer.put(value, 0, value.length);
+ buffer.flip();
+ return buffer.getLong();
+ })
+ .collect(Collectors.toList());
+ }
+
+ private static Properties getKafkaClientConfiguration() {
+ final Properties standardProps = new Properties();
+ standardProps.put("bootstrap.servers", KAFKA_CONTAINER.getBootstrapServers());
+ standardProps.put("group.id", UUID.randomUUID().toString());
+ standardProps.put("enable.auto.commit", false);
+ standardProps.put("auto.offset.reset", "earliest");
+ standardProps.put("max.partition.fetch.bytes", 256);
+ standardProps.put("zookeeper.session.timeout.ms", ZK_TIMEOUT_MILLIS);
+ standardProps.put("zookeeper.connection.timeout.ms", ZK_TIMEOUT_MILLIS);
+ return standardProps;
+ }
+
+ private Consumer<byte[], byte[]> createTestConsumer(String topic) {
+ final Properties consumerConfig = new Properties();
+ consumerConfig.putAll(getKafkaClientConfiguration());
+ consumerConfig.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ consumerConfig.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ consumerConfig.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ final KafkaConsumer<byte[], byte[]> kafkaConsumer = new KafkaConsumer<>(consumerConfig);
+ kafkaConsumer.subscribe(Collections.singletonList(topic));
+ return kafkaConsumer;
+ }
+
+ private void createTestTopic(String topic, int numPartitions, short replicationFactor)
+ throws ExecutionException, InterruptedException, TimeoutException {
+ Map<String, Object> properties = new HashMap<>();
+ properties.put(
+ CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
+ KAFKA_CONTAINER.getBootstrapServers());
+ try (AdminClient admin = AdminClient.create(properties)) {
+ final CreateTopicsResult result =
+ admin.createTopics(
+ Collections.singletonList(
+ new NewTopic(topic, numPartitions, replicationFactor)));
+ result.all().get(1, TimeUnit.MINUTES);
+ }
+ }
+
+ private void deleteTestTopic(String topic)
+ throws ExecutionException, InterruptedException, TimeoutException {
+ Map<String, Object> properties = new HashMap<>();
+ properties.put(
+ CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
+ KAFKA_CONTAINER.getBootstrapServers());
+ try (AdminClient admin = AdminClient.create(properties)) {
+ final DeleteTopicsResult result = admin.deleteTopics(Collections.singletonList(topic));
+ result.all().get(1, TimeUnit.MINUTES);
+ }
+ }
+
+ private List<ConsumerRecord<byte[], byte[]>> drainAllRecordsFromTopic(String topic) {
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords = new ArrayList<>();
+ try (Consumer<byte[], byte[]> consumer = createTestConsumer(topic)) {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ // Drain the kafka topic till all records are consumed
+ while (!records.isEmpty()) {
+ records.records(topic).forEach(collectedRecords::add);
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ }
+ return collectedRecords;
+ }
+
+ private static class RecordSerializer implements KafkaRecordSerializationSchema<Long> {
+
+ private final String topic;
+
+ public RecordSerializer(String topic) {
+ this.topic = topic;
+ }
+
+ @Override
+ public ProducerRecord<byte[], byte[]> serialize(
+ Long element, KafkaSinkContext context, Long timestamp) {
+ final ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ buffer.putLong(element);
+ return new ProducerRecord<>(topic, 0, null, null, buffer.array());
+ }
+ }
+
+ private static class FailAsyncCheckpointMapper
+ implements MapFunction<Long, Long>, CheckpointedFunction {
+ private static final ListStateDescriptor<Integer> stateDescriptor =
+ new ListStateDescriptor<>("test-state", new SlowSerializer());
+ private int failAfterCheckpoint;
+
+ private ListState<Integer> state;
+
+ public FailAsyncCheckpointMapper(int failAfterCheckpoint) {
+ this.failAfterCheckpoint = failAfterCheckpoint;
+ }
+
+ @Override
+ public Long map(Long value) throws Exception {
+ Thread.sleep(100);
+ return value;
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ state.clear();
+ if (failAfterCheckpoint <= 0) {
+ // Trigger a failure in the serializer
+ state.add(-1);
+ } else {
+ state.add(1);
+ }
+ failAfterCheckpoint--;
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ state = context.getOperatorStateStore().getListState(stateDescriptor);
+ }
+ }
+
+ private static class SlowSerializer extends TypeSerializerSingleton<Integer> {
+
+ @Override
+ public boolean isImmutableType() {
+ return false;
+ }
+
+ @Override
+ public Integer createInstance() {
+ return 1;
+ }
+
+ @Override
+ public Integer copy(Integer from) {
+ return from;
+ }
+
+ @Override
+ public Integer copy(Integer from, Integer reuse) {
+ return from;
+ }
+
+ @Override
+ public int getLength() {
+ return 0;
+ }
+
+ @Override
+ public void serialize(Integer record, DataOutputView target) throws IOException {
+ if (record != -1) {
+ return;
+ }
+ throw new RuntimeException("Expected failure during async checkpoint phase");
+ }
+
+ @Override
+ public Integer deserialize(DataInputView source) throws IOException {
+ return 1;
+ }
+
+ @Override
+ public Integer deserialize(Integer reuse, DataInputView source) throws IOException {
+ return 1;
+ }
+
+ @Override
+ public void copy(DataInputView source, DataOutputView target) throws IOException {}
+
+ @Override
+ public TypeSerializerSnapshot<Integer> snapshotConfiguration() {
+ return new SlowSerializerSnapshot();
+ }
+ }
+
+ /** Snapshot used in {@link FailAsyncCheckpointMapper}. */
+ public static class SlowSerializerSnapshot extends SimpleTypeSerializerSnapshot<Integer> {
+ public SlowSerializerSnapshot() {
+ super(SlowSerializer::new);
+ }
+ }
+
+ /** Fails after a checkpoint is taken and the next record was emitted. */
+ private static class FailingCheckpointMapper
+ implements MapFunction<Long, Long>, CheckpointListener, CheckpointedFunction {
+
+ private final SharedReference<AtomicBoolean> failed;
+ private final SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ private volatile long lastSeenRecord;
+ private volatile long checkpointedRecord;
+ private volatile long lastCheckpointId = 0;
+ private final AtomicInteger emittedBetweenCheckpoint = new AtomicInteger(0);
+
+ FailingCheckpointMapper(
+ SharedReference<AtomicBoolean> failed,
+ SharedReference<AtomicLong> lastCheckpointedRecord) {
+ this.failed = failed;
+ this.lastCheckpointedRecord = lastCheckpointedRecord;
+ }
+
+ @Override
+ public Long map(Long value) throws Exception {
+ lastSeenRecord = value;
+ if (lastCheckpointId >= 1
+ && emittedBetweenCheckpoint.get() > 0
+ && !failed.get().get()) {
+ failed.get().set(true);
+ throw new RuntimeException("Planned exception.");
+ }
+ Thread.sleep(50);
+ emittedBetweenCheckpoint.incrementAndGet();
+ return value;
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ lastCheckpointId = checkpointId;
+ emittedBetweenCheckpoint.set(0);
+ lastCheckpointedRecord.get().set(checkpointedRecord);
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ checkpointedRecord = lastSeenRecord;
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {}
+ }
+
+ private void checkProducerLeak() {
+ for (Thread t : Thread.getAllStackTraces().keySet()) {
+ if (t.getName().contains("kafka-producer-network-thread")) {
+ fail("Detected producer leak. Thread name: " + t.getName());
+ }
+ }
+ }
+
+ /**
+ * Exposes information about how man records have been emitted overall and at the beginning of a
+ * checkpoint.
+ */
+ private static final class InfiniteIntegerSource
Review comment:
Can you just use a `env.fromSequence(0, Long.MAX_VALUE)` with a chained map that implements this functionality? You are making our future lives harder :/
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriterITCase.java
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import java.time.Duration;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.contains;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for the standalone KafkaWriter. */
+@RunWith(Parameterized.class)
+public class KafkaWriterITCase {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriterITCase.class);
+ private static final Slf4jLogConsumer LOG_CONSUMER = new Slf4jLogConsumer(LOG);
+ private static final String INTER_CONTAINER_KAFKA_ALIAS = "kafka";
+ private static final Network NETWORK = Network.newNetwork();
+
+ private final DeliveryGuarantee guarantee;
+
+ private MockMetricGroup metricGroup;
+
+ @ClassRule
+ public static final KafkaContainer KAFKA_CONTAINER =
+ new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.5.2"))
+ .withEmbeddedZookeeper()
+ .withEnv(
+ ImmutableMap.of(
+ "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR",
+ "1",
+ "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
+ String.valueOf(Duration.ofHours(2).toMillis()),
+ "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR",
+ "1",
+ "KAFKA_MIN_INSYNC_REPLICAS",
+ "1"))
+ .withNetwork(NETWORK)
+ .withLogConsumer(LOG_CONSUMER)
+ .withNetworkAliases(INTER_CONTAINER_KAFKA_ALIAS);
+
+ @Before
+ public void setUp() {
+ metricGroup = new MockMetricGroup();
+ }
+
+ @Parameterized.Parameters
+ public static List<DeliveryGuarantee> guarantees() {
+ return ImmutableList.of(
+ DeliveryGuarantee.NONE,
+ DeliveryGuarantee.AT_LEAST_ONCE,
+ DeliveryGuarantee.EXACTLY_ONCE);
+ }
+
+ public KafkaWriterITCase(DeliveryGuarantee guarantee) {
+ this.guarantee = guarantee;
+ }
+
+ @Test
+ public void testRegisterMetrics() throws Exception {
+ try (final KafkaWriter<Integer> ignored =
+ createWriterWithConfiguration(getKafkaClientConfiguration())) {
+ assertThat(metricGroup.addedGroups, contains("KafkaProducer"));
+ assertFalse(metricGroup.registeredGauges.isEmpty());
+ }
+ }
+
+ @Test
+ public void testNotRegisterMetrics() throws Exception {
+ final Properties config = getKafkaClientConfiguration();
+ config.put("flink.disable-metrics", "true");
+ try (final KafkaWriter<Integer> ignored = createWriterWithConfiguration(config)) {
+ assertTrue(metricGroup.addedGroups.isEmpty());
+ assertTrue(metricGroup.registeredGauges.isEmpty());
+ }
+ }
+
+ private KafkaWriter<Integer> createWriterWithConfiguration(Properties config) {
+ return new KafkaWriter<>(
+ guarantee,
+ config,
+ "test-prefix",
+ new SinkInitContext(metricGroup),
+ new DummyRecordSerializer(),
+ new DummySchemaContext(),
+ ImmutableList.of());
+ }
+
+ private static Properties getKafkaClientConfiguration() {
+ final Properties standardProps = new Properties();
+ standardProps.put("bootstrap.servers", KAFKA_CONTAINER.getBootstrapServers());
+ standardProps.put("group.id", "kafkaWriter-tests");
+ standardProps.put("enable.auto.commit", false);
+ standardProps.put("key.serializer", ByteArraySerializer.class.getName());
+ standardProps.put("value.serializer", ByteArraySerializer.class.getName());
+ standardProps.put("auto.offset.reset", "earliest");
+ return standardProps;
+ }
+
+ private static class MockMetricGroup implements MetricGroup {
Review comment:
Can you check if we can use the `MetricListener`?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
Review comment:
Please add unit test, especially to cover a prefix with the delimiter (code looks good) ;)
Also please avoid the costly `split` and use Guava's Splitter instead.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaCommitter.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.errors.InvalidTxnStateException;
+import org.apache.kafka.common.errors.ProducerFencedException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Committer implementation for {@link KafkaSink}
+ *
+ * <p>The committer is responsible to finalize the Kafka transactions by committing them.
+ */
+class KafkaCommitter implements Committer<KafkaCommittable> {
+
+ private final Properties kafkaProducerConfig;
+
+ KafkaCommitter(Properties kafkaProducerConfig) {
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ }
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaCommitter.class);
+
+ @Override
+ public List<KafkaCommittable> commit(List<KafkaCommittable> committables) throws IOException {
+ committables.forEach(this::commitTransaction);
+ return Collections.emptyList();
+ }
+
+ @Override
+ public void close() throws Exception {}
+
+ private void commitTransaction(KafkaCommittable committable) {
+ final String transactionalId = committable.getTransactionalId();
+ LOG.debug("Committing Kafka transaction {}", transactionalId);
+ try (FlinkKafkaInternalProducer<?, ?> producer =
+ new FlinkKafkaInternalProducer<>(createKafkaProducerConfig(transactionalId))) {
+ producer.resumeTransaction(committable.getProducerId(), committable.getEpoch());
+ producer.commitTransaction();
+ } catch (InvalidTxnStateException | ProducerFencedException e) {
+ // That means we have committed this transaction before.
+ LOG.warn(
+ "Encountered error {} while recovering transaction {}. "
+ + "Presumably this transaction has been already committed before",
+ e,
+ committable);
Review comment:
Upon re-reading, I'm assuming that it actually fails elsewhere and he just sees the warning over and over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684070554
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
Review comment:
Let's do this as a followup 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684076421
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
Review comment:
renamed it to `producerRecord`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684129649
##########
File path: flink-tests/src/test/java/org/apache/flink/test/util/TestUtils.java
##########
@@ -84,4 +92,32 @@ public static void waitUntilJobInitializationFinished(
() -> clusterClient.requestJobResult(id).get(),
userCodeClassloader);
}
+
+ public static File getMostRecentCompletedCheckpoint(File checkpointDir) throws IOException {
+ return Files.find(checkpointDir.toPath(), 2, TestUtils::isCompletedCheckpoint)
Review comment:
FYI there is a bug here https://issues.apache.org/jira/browse/FLINK-23647. But we would fix it with that ticket.
##########
File path: flink-core/src/main/java/org/apache/flink/api/connector/sink/InitContextInitializationContextAdapter.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.connector.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import java.util.function.Function;
+
+/**
+ * Adapter between {@link Sink.InitContext} and {@link SerializationSchema.InitializationContext}.
+ */
+public class InitContextInitializationContextAdapter
Review comment:
I see that this is implemented similarly to `RuntimeContextSerializationInitializationContextAdapter` but has the same flaws.
In general, when implementing a small class it should not rely on a big class to be injected. That makes reuse and testing much harder. In this case, it would be much better to just pass the `UserCodeClassLoader` and the `MetricGroup` directly and have no dependency to the `InitContext`.
Now the metric group should only be added when it's actually needed, so here I would rather use a `Supplier<MetricGroup>` (it doesn't make sense that the `InitContext` is passed to this Adapter and then passed back to the `mapMetricGroup`; the `InitContext` is by definition available on call-site)
So signature should be
```
public InitContextInitializationContextAdapter(
Supplier<MetricGroup> metricGroupSupplier, UserCodeClassLoader userCodeClassLoader)
```
Finally we should cache the result from the Supplier most easily by using `Supplier#memoize` of Guava. I'd probably wrap the ctor parameter before assigning it to the field.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/TransactionalIdFactory.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.base.Splitter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+class TransactionalIdFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TransactionalIdFactory.class);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
Review comment:
Fun fact: java compiler generates this code for you if you just use simple string concatenation. You just need StringBuilder for advanced use cases, such as loops, nowadays.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/TransactionalIdFactory.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.base.Splitter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+class TransactionalIdFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TransactionalIdFactory.class);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
Review comment:
Here we could also have an instantiable `TransactionalIdFactory` with constant transactionalIdPrefix and subtaskId. You could then have a pre-computed subtask prefix consisting of
```
prefix = sb.append(transactionalIdPrefix)
.append(TRANSACTIONAL_ID_DELIMITER)
.append(subtaskId)
.append(TRANSACTIONAL_ID_DELIMITER)
```
Then this method would just `return prefix + offset`;
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/TransactionalIdFactory.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.base.Splitter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+class TransactionalIdFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TransactionalIdFactory.class);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
Review comment:
This could be confused with Kafka offset. Maybe use seq number?
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkITCase.java
##########
@@ -0,0 +1,634 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.StateBackendOptions;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.test.util.TestUtils;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.testutils.junit.SharedReference;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.CreateTopicsResult;
+import org.apache.kafka.clients.admin.DeleteTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.LongStream;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+/** Tests for using KafkaSink writing to a Kafka cluster. */
+public class KafkaSinkITCase extends TestLogger {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkITCase.class);
+ private static final Slf4jLogConsumer LOG_CONSUMER = new Slf4jLogConsumer(LOG);
+ private static final String INTER_CONTAINER_KAFKA_ALIAS = "kafka";
+ private static final Network NETWORK = Network.newNetwork();
+ private static final int ZK_TIMEOUT_MILLIS = 30000;
+ private static final short TOPIC_REPLICATION_FACTOR = 1;
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+
+ private String topic;
+ private SharedReference<AtomicLong> emittedRecordsCount;
+ private SharedReference<AtomicLong> emittedRecordsWithCheckpoint;
+ private SharedReference<AtomicBoolean> failed;
+ private SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ @ClassRule
+ public static final KafkaContainer KAFKA_CONTAINER =
+ new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.5.2"))
+ .withEmbeddedZookeeper()
+ .withEnv(
+ ImmutableMap.of(
+ "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR",
+ "1",
+ "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
+ String.valueOf(Duration.ofHours(2).toMillis()),
+ "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR",
+ "1",
+ "KAFKA_MIN_INSYNC_REPLICAS",
+ "1"))
+ .withNetwork(NETWORK)
+ .withLogConsumer(LOG_CONSUMER)
+ .withNetworkAliases(INTER_CONTAINER_KAFKA_ALIAS);
+
+ @Rule public final SharedObjects sharedObjects = SharedObjects.create();
+
+ @Rule public final TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void setUp() throws ExecutionException, InterruptedException, TimeoutException {
+ emittedRecordsCount = sharedObjects.add(new AtomicLong());
+ emittedRecordsWithCheckpoint = sharedObjects.add(new AtomicLong());
+ failed = sharedObjects.add(new AtomicBoolean(false));
+ lastCheckpointedRecord = sharedObjects.add(new AtomicLong(0));
+ topic = UUID.randomUUID().toString();
+ createTestTopic(topic, 1, TOPIC_REPLICATION_FACTOR);
+ }
+
+ @After
+ public void tearDown() throws ExecutionException, InterruptedException, TimeoutException {
+ deleteTestTopic(topic);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithAtLeastOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.AT_LEAST_ONCE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithNoneGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.NONE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithExactlyOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.EXACTLY_ONCE, emittedRecordsWithCheckpoint);
+ }
+
+ @Test
+ public void testRecoveryWithAtLeastOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.AT_LEAST_ONCE,
+ (records) ->
+ assertThat(records, hasItems(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L)));
+ }
+
+ @Test
+ public void testRecoveryWithExactlyOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.EXACTLY_ONCE,
+ (records) ->
+ assertEquals(
+ records,
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList())));
+ }
+
+ @Test
+ public void testAbortTransactionsOfPendingCheckpointsAfterFailure() throws Exception {
+ // Run a first job failing during the async phase of a checkpoint to leave some
+ // lingering transactions
+ final Configuration config = new Configuration();
+ config.setString(StateBackendOptions.STATE_BACKEND, "filesystem");
+ final File checkpointDir = temp.newFolder();
+ config.setString(
+ CheckpointingOptions.CHECKPOINTS_DIRECTORY, checkpointDir.toURI().toString());
+ config.set(
+ ExecutionCheckpointingOptions.EXTERNALIZED_CHECKPOINT,
+ CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
+ config.set(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 2);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(1), config, "firstPrefix");
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ final File completedCheckpoint = TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
+
+ config.set(SavepointConfigOptions.SAVEPOINT_PATH, completedCheckpoint.toURI().toString());
+
+ // Run a second job which aborts all lingering transactions and new consumer should
+ // immediately see the newly written records
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, "newPrefix");
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ @Test
+ public void testAbortTransactionsAfterScaleInBeforeFirstCheckpoint() throws Exception {
+ // Run a first job opening 5 transactions one per subtask and fail in async checkpoint phase
+ final Configuration config = new Configuration();
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 5);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(0), config, null);
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ assertTrue(deserializeValues(drainAllRecordsFromTopic(topic)).isEmpty());
+
+ // Second job aborts all transactions from previous runs with higher parallelism
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 1);
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, null);
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ private void executeWithMapper(
+ MapFunction<Long, Long> mapper,
+ Configuration config,
+ @Nullable String transactionalIdPrefix)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment(config);
+ env.enableCheckpointing(100L);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ final DataStreamSource<Long> source = env.fromSequence(1, 10);
+ final DataStream<Long> stream = source.map(mapper);
+ final KafkaSinkBuilder<Long> builder =
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic));
+ if (transactionalIdPrefix == null) {
+ transactionalIdPrefix = "kafka-sink";
+ }
+ builder.setTransactionalIdPrefix(transactionalIdPrefix);
+ stream.sinkTo(builder.build());
+ env.execute();
+ checkProducerLeak();
+ }
+
+ private void testRecoveryWithAssertion(
+ DeliveryGuarantee guarantee, java.util.function.Consumer<List<Long>> recordsAssertion)
+ throws Exception {
Review comment:
FYI, there is another pattern that can be used to implement such a wrapping setup/cleanup code.
```
AutoCloseableResult testRecovery(DeliveryGuarantee guarantee) {
// execute common code
result = // fetch result
AutoCloseable after = () -> { // after code };
return wrap(result, after);
}
```
You can then use the return value in a try-and-resource and add all your assertions in the block. It has huge benefits over your pattern when you have checked exceptions and often is easier on the eye when the auto-formatter went over it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0f0ab5854917b30e283dd7195690dd46ee180521 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317)
* 6287ea41e01d215a3974dca0ae037537a7f3d4bb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
}, {
"hash" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"triggerType" : "PUSH"
}, {
"hash" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681",
"triggerID" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"triggerType" : "PUSH"
}, {
"hash" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"triggerType" : "PUSH"
}, {
"hash" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21692",
"triggerID" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"triggerType" : "PUSH"
}, {
"hash" : "483a1a6e982e43b538f13152d41b71144143e444",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21701",
"triggerID" : "483a1a6e982e43b538f13152d41b71144143e444",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 5f636910791eb5ddaa6b9b210c3a4b3230fc60bd UNKNOWN
* 96995fe1f26d7a08e91308409d2c66ec32ec8819 UNKNOWN
* 483a1a6e982e43b538f13152d41b71144143e444 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21701)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r683310573
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
+
+ private static byte readTransactionState(ByteBuffer buffer) {
+ // producerId
+ buffer.getLong();
+ // epoch
+ buffer.getShort();
+ // transactionTimeout
+ buffer.getInt();
+ // statusKey
+ return buffer.get();
+ }
+
+ enum TransactionState {
+ Empty(Byte.parseByte("0")),
Review comment:
I am not convinced by all options 😄 Do you mind if I leave it as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25fb6f9a84ab137be93a04bfad41c4ab021a3b4b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464)
* 1e41696e78086dda1970f6e50e3125d0271c056b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 5916417a1266f4af34d387dd2e61a10673a1a9fe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
}, {
"hash" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"triggerType" : "PUSH"
}, {
"hash" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681",
"triggerID" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"triggerType" : "PUSH"
}, {
"hash" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"triggerType" : "PUSH"
}, {
"hash" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21692",
"triggerID" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"triggerType" : "PUSH"
}, {
"hash" : "483a1a6e982e43b538f13152d41b71144143e444",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "483a1a6e982e43b538f13152d41b71144143e444",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* e8c6ec06f2263f5704ccf597494c43a1bd0e5893 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664)
* 5f636910791eb5ddaa6b9b210c3a4b3230fc60bd UNKNOWN
* 055d0fd83c72f141a1bd2d8a4bc0e30bf637687f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681)
* 96995fe1f26d7a08e91308409d2c66ec32ec8819 UNKNOWN
* 2d1d736dc15e4b96b13a917ecfdd9716d8d48be4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21692)
* 483a1a6e982e43b538f13152d41b71144143e444 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684130557
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriterState.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class KafkaWriterState {
+ private final int subtaskId;
+ private final long transactionalIdOffset;
+ private final String transactionalIdPrefix;
Review comment:
Yeah downscaling before checkpoint case is definitely a problem, I can update the doc string to hint that the prefix should remain stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e41696e78086dda1970f6e50e3125d0271c056b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494)
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 77d980b81bcd178befe0a1f3762234a2e5b79222 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520)
* 3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684080102
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
+ recordSerializer.serialize(element, kafkaSinkContext, context.timestamp());
+ pendingRecords.incrementAndGet();
+ currentProducer.send(record, deliveryCallback);
+ }
+
+ @Override
+ public List<KafkaCommittable> prepareCommit(boolean flush) throws IOException {
+ flushRecords(flush);
+ if (!flush) {
+ currentProducer = beginTransaction();
+ }
+ final List<KafkaCommittable> committables = commit();
+ producers.add(currentProducer);
+ return committables;
+ }
+
+ @Override
+ public List<KafkaWriterState> snapshotState() throws IOException {
+ return ImmutableList.of(kafkaWriterState);
+ }
+
+ @Override
+ public void close() throws Exception {
+ currentProducer.close(Duration.ZERO);
+ }
+
+ private KafkaWriterState recoverAndInitializeState(List<KafkaWriterState> recoveredStates) {
+ final int subtaskId = kafkaSinkContext.getParallelInstanceId();
+ if (recoveredStates.isEmpty()) {
+ final KafkaWriterState state =
+ new KafkaWriterState(transactionalIdPrefix, subtaskId, 0);
+ abortTransactions(getTransactionsToAbort(state, new ArrayList<>()));
+ return state;
+ }
+ final Map<Integer, KafkaWriterState> taskOffsetMapping =
+ recoveredStates.stream()
+ .collect(
+ Collectors.toMap(
+ KafkaWriterState::getSubtaskId, Function.identity()));
+ if (!taskOffsetMapping.containsKey(subtaskId)) {
+ throw new IllegalStateException(
+ "It is expected that state from previous executions is distributed to the same subtask id.");
+ }
+ final KafkaWriterState lastState = taskOffsetMapping.get(subtaskId);
+ taskOffsetMapping.remove(subtaskId);
+ abortTransactions(
+ getTransactionsToAbort(lastState, new ArrayList<>(taskOffsetMapping.values())));
+ if (!lastState.getTransactionalIdPrefix().equals(transactionalIdPrefix)) {
+ LOG.warn(
+ "Transactional id prefix from previous execution {} has changed to {}.",
+ lastState.getTransactionalIdPrefix(),
+ transactionalIdPrefix);
+ return new KafkaWriterState(transactionalIdPrefix, subtaskId, 0);
+ }
+ return new KafkaWriterState(
+ transactionalIdPrefix, subtaskId, lastState.getTransactionalIdOffset());
+ }
+
+ private void abortTransactions(Set<String> transactionsToAbort) {
+ transactionsToAbort.forEach(
+ transaction -> {
+ // don't mess with the original configuration or any other
+ // properties of the
+ // original object
+ // -> create an internal kafka producer on our own and do not rely
+ // on
+ // initTransactionalProducer().
+ final Properties myConfig = new Properties();
+ myConfig.putAll(kafkaProducerConfig);
+ myConfig.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transaction);
+ LOG.info("Aborting Kafka transaction {}.", transaction);
+ FlinkKafkaInternalProducer<byte[], byte[]> kafkaProducer = null;
Review comment:
Hmm, currently the KafkaComitter leverages the try-with-resources logic because it is important to block until the transaction is finished. By changing this behavior we would need to change the committer try-with-resource block. I am not sure we gain much here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25fb6f9a84ab137be93a04bfad41c4ab021a3b4b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464)
* 1e41696e78086dda1970f6e50e3125d0271c056b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494)
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af9dd2b341dfe555224f46d3e14a7a83c1b5db04 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0f0ab5854917b30e283dd7195690dd46ee180521 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22e7bb8757f57435a056f4dd720659c5a3207339 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310)
* 0f0ab5854917b30e283dd7195690dd46ee180521 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0f0ab5854917b30e283dd7195690dd46ee180521 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317)
* 6287ea41e01d215a3974dca0ae037537a7f3d4bb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
}, {
"hash" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"triggerType" : "PUSH"
}, {
"hash" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681",
"triggerID" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"triggerType" : "PUSH"
}, {
"hash" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* e8c6ec06f2263f5704ccf597494c43a1bd0e5893 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664)
* 5f636910791eb5ddaa6b9b210c3a4b3230fc60bd UNKNOWN
* 055d0fd83c72f141a1bd2d8a4bc0e30bf637687f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681)
* 96995fe1f26d7a08e91308409d2c66ec32ec8819 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
}, {
"hash" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"triggerType" : "PUSH"
}, {
"hash" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681",
"triggerID" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"triggerType" : "PUSH"
}, {
"hash" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"triggerType" : "PUSH"
}, {
"hash" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* e8c6ec06f2263f5704ccf597494c43a1bd0e5893 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664)
* 5f636910791eb5ddaa6b9b210c3a4b3230fc60bd UNKNOWN
* 055d0fd83c72f141a1bd2d8a4bc0e30bf637687f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681)
* 96995fe1f26d7a08e91308409d2c66ec32ec8819 UNKNOWN
* 2d1d736dc15e4b96b13a917ecfdd9716d8d48be4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684152881
##########
File path: flink-tests/src/test/java/org/apache/flink/test/util/TestUtils.java
##########
@@ -84,4 +92,32 @@ public static void waitUntilJobInitializationFinished(
() -> clusterClient.requestJobResult(id).get(),
userCodeClassloader);
}
+
+ public static File getMostRecentCompletedCheckpoint(File checkpointDir) throws IOException {
+ return Files.find(checkpointDir.toPath(), 2, TestUtils::isCompletedCheckpoint)
Review comment:
Which ticket do you mean by `that` ticket?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r683305461
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
+ * #setDeliverGuarantee(DeliveryGuarantee)} but keep in mind when using {@link
+ * DeliveryGuarantee#EXACTLY_ONCE} one must set the transactionalIdPrefix {@link
+ * #setTransactionalIdPrefix(String)}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+public class KafkaSinkBuilder<IN> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkBuilder.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private String transactionalIdPrefix = "kafka-sink";
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+ private String bootstrapServers;
+
+ /**
+ * Sets the wanted the {@link DeliveryGuarantee}. The default delivery guarantee is {@link
+ * #deliveryGuarantee}.
+ *
+ * @param deliveryGuarantee
+ * @return {@link KafkaSinkBuilder}
+ */
+ KafkaSinkBuilder<IN> setDeliverGuarantee(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "semantic");
Review comment:
I will definitely leave the second parameter out next time but now I want to keep it consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684156074
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/TransactionalIdFactory.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.base.Splitter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+class TransactionalIdFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TransactionalIdFactory.class);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
Review comment:
I renamed it to `checkpointOffset`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-892702719
@AHeise thanks for your review. I have addressed all your comments, please have another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6287ea41e01d215a3974dca0ae037537a7f3d4bb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 5916417a1266f4af34d387dd2e61a10673a1a9fe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25fb6f9a84ab137be93a04bfad41c4ab021a3b4b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 864808edf90294e0fa1386c321ae752a7e076b97 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639)
* e8c6ec06f2263f5704ccf597494c43a1bd0e5893 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531)
* 5916417a1266f4af34d387dd2e61a10673a1a9fe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a4a37916fdd2d49a7682b2d90da22e5ed398eae5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429)
* 25fb6f9a84ab137be93a04bfad41c4ab021a3b4b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e41696e78086dda1970f6e50e3125d0271c056b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494)
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 77d980b81bcd178befe0a1f3762234a2e5b79222 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520)
* 3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531)
* 5916417a1266f4af34d387dd2e61a10673a1a9fe UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise merged pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
AHeise merged pull request #16676:
URL: https://github.com/apache/flink/pull/16676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r683442480
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
Review comment:
I think we already had this discussion ;) Let's clarify it offline I am pretty sure the current solution also covers your described case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r683265825
##########
File path: flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointTestBase.java
##########
@@ -165,35 +161,11 @@ protected File execute(UnalignedSettings settings) throws Exception {
miniCluster.after();
}
if (settings.generateCheckpoint) {
- return Files.find(checkpointDir.toPath(), 2, this::isCompletedCheckpoint)
- .max(Comparator.comparing(Path::toString))
- .map(Path::toFile)
- .orElseThrow(() -> new IllegalStateException("Cannot generate checkpoint"));
+ return TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
Review comment:
This is not easily possible because I cannot add the `TestUtils` dependency because it would introduce a cycling dependency. Therefore I left it, for now, to not spending more time entangling the dependencies.
##########
File path: flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointTestBase.java
##########
@@ -165,35 +161,11 @@ protected File execute(UnalignedSettings settings) throws Exception {
miniCluster.after();
}
if (settings.generateCheckpoint) {
- return Files.find(checkpointDir.toPath(), 2, this::isCompletedCheckpoint)
- .max(Comparator.comparing(Path::toString))
- .map(Path::toFile)
- .orElseThrow(() -> new IllegalStateException("Cannot generate checkpoint"));
+ return TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
Review comment:
This is not easily possible because I cannot add the `TestUtils` dependency because it would introduce a cycling dependency. Therefore I left it, for now, to not spending more time disentangling the dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 864808edf90294e0fa1386c321ae752a7e076b97 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684085369
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriterState.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class KafkaWriterState {
+ private final int subtaskId;
+ private final long transactionalIdOffset;
+ private final String transactionalIdPrefix;
Review comment:
I have to store it here to abort transactions from previous runs. If the job is stopped and started with a new prefix the new one is used for all newly created states.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890959375
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit af9dd2b341dfe555224f46d3e14a7a83c1b5db04 (Mon Aug 02 11:45:09 UTC 2021)
**Warnings:**
* **1 pom.xml files were touched**: Check for build and licensing issues.
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r683151739
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/DefaultKafkaSinkContext.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Sink;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.common.PartitionInfo;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Context providing information to assist constructing a {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}.
+ */
+class DefaultKafkaSinkContext implements KafkaRecordSerializationSchema.KafkaSinkContext {
+
+ private final Sink.InitContext context;
+ private final Properties kafkaProducerConfig;
+
+ private int[] cachedPartitions;
+
+ public DefaultKafkaSinkContext(Sink.InitContext context, Properties kafkaProducerConfig) {
+ this.context = context;
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ }
+
+ @Override
+ public int getParallelInstanceId() {
+ return context.getSubtaskId();
+ }
+
+ @Override
+ public int getNumberOfParallelInstances() {
+ return context.getNumberOfParallelSubtasks();
+ }
+
+ @Override
Review comment:
```suggestion
@Override
public int[] getPartitionsForTopic(String topic) {
return cachedPartitions.computeIfAbsent(topic, this::fetchPartitions);
}
private int[] fetchPartitions(String topic) {
try (final Producer<?, ?> producer = new KafkaProducer<>(kafkaProducerConfig)) {
return producer.partitionsFor(topic).stream()
.mapToInt(PartitionInfo::partition)
.sorted()
.toArray();
}
}
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaCommittable.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+
+import java.util.Objects;
+
+/**
+ * This class holds the necessary information to construct a new {@link FlinkKafkaInternalProducer}
+ * to commit transactions in {@link KafkaCommitter}.
+ */
+class KafkaCommittable {
+
+ private final long producerId;
+ private final short epoch;
+ private final String transactionalId;
+
+ public KafkaCommittable(FlinkKafkaInternalProducer<byte[], byte[]> producer) {
+ this(
+ producer.getProducerId(),
+ producer.getEpoch(),
+ producer.getKafkaProducerConfig()
+ .getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG));
+ }
+
Review comment:
Nit: Please use a factory method when translating from one object to another.
See also https://www.programmersought.com/article/22326505572/
##########
File path: flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointTestBase.java
##########
@@ -165,35 +161,11 @@ protected File execute(UnalignedSettings settings) throws Exception {
miniCluster.after();
}
if (settings.generateCheckpoint) {
- return Files.find(checkpointDir.toPath(), 2, this::isCompletedCheckpoint)
- .max(Comparator.comparing(Path::toString))
- .map(Path::toFile)
- .orElseThrow(() -> new IllegalStateException("Cannot generate checkpoint"));
+ return TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
Review comment:
👍
Could you please also update CoordinatedSourceRescaleITCase?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} In this mode the KafkaSink will write all messages in
+ * a Kafka transaction that will be committed to Kafka on a checkpoint. It is very important to
+ * ensure that you use unique {@link #transactionalIdPrefix}s across your applications running
+ * on the same Kafka cluster otherwise I may happen that multiple running jobs corrupt their
+ * transactions!
Review comment:
```suggestion
* <li>{@link DeliveryGuarantee#EXACTLY_ONCE}: In this mode the KafkaSink will write all messages in
* a Kafka transaction that will be committed to Kafka on a checkpoint. Thus, if the consumer reads only committed data (see Kafka consumer config isolation.level), no duplicates will be seen in case of a Flink restart. However, this delays record writing effectively until a checkpoint is written, so adjust the checkpoint duration accordingly.
Please ensure that you use unique {@link #transactionalIdPrefix}s across your applications running
* on the same Kafka cluster such that multiple running jobs do not interfere in their
* transactions! Additionally, it is highly recommended to tweak Kafka transaction timeout (link) >> maximum checkpoint duration + maximum restart duration or data loss may happen when Kafka expires an uncommitted transaction.
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
+
+ private static byte readTransactionState(ByteBuffer buffer) {
+ // producerId
+ buffer.getLong();
+ // epoch
+ buffer.getShort();
+ // transactionTimeout
+ buffer.getInt();
+ // statusKey
+ return buffer.get();
+ }
Review comment:
```suggestion
private static byte readTransactionState(ByteBuffer buffer) {
return buffer.get(TRANSACTION_STATE_POS);
}
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
Review comment:
```suggestion
private void maybeAddTransaction(ConsumerRecord<byte[], byte[]> record,
Set<String> openTransactions) {
```
and do the lambda on call-site.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
+ recordSerializer.serialize(element, kafkaSinkContext, context.timestamp());
+ pendingRecords.incrementAndGet();
+ currentProducer.send(record, deliveryCallback);
+ }
+
+ @Override
+ public List<KafkaCommittable> prepareCommit(boolean flush) throws IOException {
+ flushRecords(flush);
+ if (!flush) {
+ currentProducer = beginTransaction();
+ }
+ final List<KafkaCommittable> committables = commit();
+ producers.add(currentProducer);
+ return committables;
+ }
+
+ @Override
+ public List<KafkaWriterState> snapshotState() throws IOException {
+ return ImmutableList.of(kafkaWriterState);
+ }
+
+ @Override
+ public void close() throws Exception {
+ currentProducer.close(Duration.ZERO);
+ }
+
+ private KafkaWriterState recoverAndInitializeState(List<KafkaWriterState> recoveredStates) {
+ final int subtaskId = kafkaSinkContext.getParallelInstanceId();
+ if (recoveredStates.isEmpty()) {
+ final KafkaWriterState state =
+ new KafkaWriterState(transactionalIdPrefix, subtaskId, 0);
+ abortTransactions(getTransactionsToAbort(state, new ArrayList<>()));
+ return state;
+ }
+ final Map<Integer, KafkaWriterState> taskOffsetMapping =
+ recoveredStates.stream()
+ .collect(
+ Collectors.toMap(
+ KafkaWriterState::getSubtaskId, Function.identity()));
+ if (!taskOffsetMapping.containsKey(subtaskId)) {
+ throw new IllegalStateException(
+ "It is expected that state from previous executions is distributed to the same subtask id.");
+ }
Review comment:
`checkState` also maybe add `internal error:`?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
+ * #setDeliverGuarantee(DeliveryGuarantee)} but keep in mind when using {@link
+ * DeliveryGuarantee#EXACTLY_ONCE} one must set the transactionalIdPrefix {@link
+ * #setTransactionalIdPrefix(String)}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+public class KafkaSinkBuilder<IN> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkBuilder.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private String transactionalIdPrefix = "kafka-sink";
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+ private String bootstrapServers;
+
+ /**
+ * Sets the wanted the {@link DeliveryGuarantee}. The default delivery guarantee is {@link
+ * #deliveryGuarantee}.
+ *
+ * @param deliveryGuarantee
+ * @return {@link KafkaSinkBuilder}
+ */
+ KafkaSinkBuilder<IN> setDeliverGuarantee(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "semantic");
+ return this;
+ }
+
+ /**
+ * Sets the configuration which used to instantiate all used {@link
+ * org.apache.kafka.clients.producer.KafkaProducer}.
+ *
+ * @param kafkaProducerConfig
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setKafkaProducerConfig(Properties kafkaProducerConfig) {
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ // set the producer configuration properties for kafka record key value serializers.
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG)) {
+ final long timeout = DEFAULT_KAFKA_TRANSACTION_TIMEOUT.toMillis();
+ checkState(
+ timeout < Integer.MAX_VALUE && timeout > 0,
+ "timeout does not fit into 32 bit integer");
+ kafkaProducerConfig.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, (int) timeout);
+ LOG.warn(
+ "Property [{}] not specified. Setting it to {}",
+ ProducerConfig.TRANSACTION_TIMEOUT_CONFIG,
+ DEFAULT_KAFKA_TRANSACTION_TIMEOUT);
+ }
+ return this;
+ }
+
+ /**
+ * Sets the {@link KafkaRecordSerializationSchema} that transforms incoming records to {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}s.
+ *
+ * @param recordSerializer
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setRecordSerializer(
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ ClosureCleaner.clean(
+ this.recordSerializer, ExecutionConfig.ClosureCleanerLevel.RECURSIVE, true);
+ return this;
+ }
+
+ /**
+ * Sets the prefix for all created transactionalIds if {@link DeliveryGuarantee#EXACTLY_ONCE} is
+ * configured.
+ *
+ * <p>It is mandatory to always set this value with {@link DeliveryGuarantee#EXACTLY_ONCE} to
+ * prevent corrupted transactions if multiple jobs using the KafkaSink run against the same
+ * Kafka Cluster. The default prefix is {@link #transactionalIdPrefix}.
+ *
+ * @param transactionalIdPrefix
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setTransactionalIdPrefix(String transactionalIdPrefix) {
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
Review comment:
Please add some length check. I think the maximum kafka transaction id length is 64k (please double check), so we could cap at 60k and still have some leeway.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
Review comment:
Just to be sure: this procedure would also abort all transactions from "unknown" subtasks because the checkpoint would not reflect the true previous parallelism (upscale-downscale without checkpoint), correct?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/DefaultKafkaSinkContext.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Sink;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.common.PartitionInfo;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Context providing information to assist constructing a {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}.
+ */
+class DefaultKafkaSinkContext implements KafkaRecordSerializationSchema.KafkaSinkContext {
+
+ private final Sink.InitContext context;
+ private final Properties kafkaProducerConfig;
+
+ private int[] cachedPartitions;
Review comment:
```suggestion
private final Map<String, int[]> cachedPartitions = new HashMap<>();
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/FlinkKafkaInternalProducer.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.TransactionalRequestResult;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Properties;
+
+/**
+ * A {@link KafkaProducer} that exposes private fields to allow resume producing from a given state.
+ */
+class FlinkKafkaInternalProducer<K, V> extends KafkaProducer<K, V> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkKafkaInternalProducer.class);
+ private static final String TRANSACTION_MANAGER_STATE_ENUM =
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State";
+
+ private final Properties kafkaProducerConfig;
+ @Nullable private final String transactionalId;
+
+ public FlinkKafkaInternalProducer(Properties properties) {
+ super(properties);
+ this.kafkaProducerConfig = properties;
+ this.transactionalId = properties.getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ if (transactionalId != null) {
+ flushNewPartitions();
+ }
+ }
+
+ public Properties getKafkaProducerConfig() {
+ return kafkaProducerConfig;
+ }
+
+ public short getEpoch() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (short) getField(producerIdAndEpoch, "epoch");
+ }
+
+ public long getProducerId() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (long) getField(producerIdAndEpoch, "producerId");
+ }
+
+ /**
+ * Besides committing {@link org.apache.kafka.clients.producer.KafkaProducer#commitTransaction}
+ * is also adding new partitions to the transaction. flushNewPartitions method is moving this
+ * logic to pre-commit/flush, to make resumeTransaction simpler. Otherwise resumeTransaction
+ * would require to restore state of the not yet added/"in-flight" partitions.
+ */
+ private void flushNewPartitions() {
+ LOG.info("Flushing new partitions");
+ TransactionalRequestResult result = enqueueNewPartitions();
+ Object sender = getField("sender");
+ invoke(sender, "wakeup");
+ result.await();
+ }
+
+ /**
+ * Enqueues new transactions at the transaction manager and returns a {@link
+ * TransactionalRequestResult} that allows waiting on them.
+ *
+ * <p>If there are no new transactions we return a {@link TransactionalRequestResult} that is
+ * already done.
+ */
+ private TransactionalRequestResult enqueueNewPartitions() {
+ Object transactionManager = getField("transactionManager");
+ synchronized (transactionManager) {
+ Object newPartitionsInTransaction =
+ getField(transactionManager, "newPartitionsInTransaction");
+ Object newPartitionsInTransactionIsEmpty =
+ invoke(newPartitionsInTransaction, "isEmpty");
+ TransactionalRequestResult result;
+ if (newPartitionsInTransactionIsEmpty instanceof Boolean
+ && !((Boolean) newPartitionsInTransactionIsEmpty)) {
+ Object txnRequestHandler =
+ invoke(transactionManager, "addPartitionsToTransactionHandler");
+ invoke(
+ transactionManager,
+ "enqueueRequest",
+ new Class[] {txnRequestHandler.getClass().getSuperclass()},
+ new Object[] {txnRequestHandler});
+ result =
+ (TransactionalRequestResult)
+ getField(
+ txnRequestHandler,
+ txnRequestHandler.getClass().getSuperclass(),
+ "result");
+ } else {
+ // we don't have an operation but this operation string is also used in
+ // addPartitionsToTransactionHandler.
+ result = new TransactionalRequestResult("AddPartitionsToTxn");
+ result.done();
+ }
+ return result;
+ }
+ }
+
+ private static Object invoke(Object object, String methodName, Object... args) {
+ Class<?>[] argTypes = new Class[args.length];
+ for (int i = 0; i < args.length; i++) {
+ argTypes[i] = args[i].getClass();
+ }
+ return invoke(object, methodName, argTypes, args);
+ }
+
+ private static Object invoke(
+ Object object, String methodName, Class<?>[] argTypes, Object[] args) {
+ try {
+ Method method = object.getClass().getDeclaredMethod(methodName, argTypes);
+ method.setAccessible(true);
+ return method.invoke(object, args);
+ } catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
Review comment:
We should switch to Guava's Invokable here.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaCommittableSerializer.java
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+
+class KafkaCommittableSerializer implements SimpleVersionedSerializer<KafkaCommittable> {
+
+ @Override
+ public int getVersion() {
+ return 1;
+ }
+
+ @Override
+ public byte[] serialize(KafkaCommittable state) throws IOException {
+ try (final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ final DataOutputStream out = new DataOutputStream(baos)) {
+ out.writeShort(state.getEpoch());
+ out.writeLong(state.getProducerId());
+ out.writeUTF(state.getTransactionalId());
Review comment:
Just FYI: `writeUTF` is limited to 64k strings, so always be alert when you see it. However, Kafka transactions have the same limit so this is fine.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
Review comment:
double negation ;)
```suggestion
* <li>{@link DeliveryGuarantee#NONE} does not provide any guarantees: messages may be lost in case of issues on the Kafka broker and messages may be duplicated in case of a Flink failure.
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaCommitter.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.errors.InvalidTxnStateException;
+import org.apache.kafka.common.errors.ProducerFencedException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Committer implementation for {@link KafkaSink}
+ *
+ * <p>The committer is responsible to finalize the Kafka transactions by committing them.
+ */
+class KafkaCommitter implements Committer<KafkaCommittable> {
+
+ private final Properties kafkaProducerConfig;
+
+ KafkaCommitter(Properties kafkaProducerConfig) {
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ }
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaCommitter.class);
+
+ @Override
+ public List<KafkaCommittable> commit(List<KafkaCommittable> committables) throws IOException {
+ committables.forEach(this::commitTransaction);
+ return Collections.emptyList();
+ }
+
+ @Override
+ public void close() throws Exception {}
+
+ private void commitTransaction(KafkaCommittable committable) {
+ final String transactionalId = committable.getTransactionalId();
+ LOG.debug("Committing Kafka transaction {}", transactionalId);
+ try (FlinkKafkaInternalProducer<?, ?> producer =
+ new FlinkKafkaInternalProducer<>(createKafkaProducerConfig(transactionalId))) {
+ producer.resumeTransaction(committable.getProducerId(), committable.getEpoch());
+ producer.commitTransaction();
+ } catch (InvalidTxnStateException | ProducerFencedException e) {
+ // That means we have committed this transaction before.
+ LOG.warn(
+ "Encountered error {} while recovering transaction {}. "
+ + "Presumably this transaction has been already committed before",
+ e,
+ committable);
Review comment:
I'm wondering if we need to do something else, see first mail in `Error using KafkaProducer EXACTLY_ONCE semantic + TaskManager Failure` on user ML.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/FlinkKafkaInternalProducer.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.TransactionalRequestResult;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Properties;
+
+/**
+ * A {@link KafkaProducer} that exposes private fields to allow resume producing from a given state.
+ */
+class FlinkKafkaInternalProducer<K, V> extends KafkaProducer<K, V> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkKafkaInternalProducer.class);
+ private static final String TRANSACTION_MANAGER_STATE_ENUM =
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State";
+
+ private final Properties kafkaProducerConfig;
+ @Nullable private final String transactionalId;
+
+ public FlinkKafkaInternalProducer(Properties properties) {
+ super(properties);
+ this.kafkaProducerConfig = properties;
+ this.transactionalId = properties.getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ if (transactionalId != null) {
+ flushNewPartitions();
+ }
+ }
+
+ public Properties getKafkaProducerConfig() {
+ return kafkaProducerConfig;
+ }
+
+ public short getEpoch() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
Review comment:
Extract constants for often used fields (lets say >2 usages).
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
Review comment:
```suggestion
* <li>{@link DeliveryGuarantee#AT_LEAST_ONCE}: the sink will wait for all outstanding records in the
* Kafka buffers to be acknowledged by the Kafka producer on a checkpoint. No messages will be lost in case of any issue with the Kafka brokers but messages may be duplicated when Flink restarts.
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/FlinkKafkaInternalProducer.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.TransactionalRequestResult;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Properties;
+
+/**
+ * A {@link KafkaProducer} that exposes private fields to allow resume producing from a given state.
+ */
+class FlinkKafkaInternalProducer<K, V> extends KafkaProducer<K, V> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkKafkaInternalProducer.class);
+ private static final String TRANSACTION_MANAGER_STATE_ENUM =
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State";
+
+ private final Properties kafkaProducerConfig;
+ @Nullable private final String transactionalId;
+
+ public FlinkKafkaInternalProducer(Properties properties) {
+ super(properties);
+ this.kafkaProducerConfig = properties;
+ this.transactionalId = properties.getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ if (transactionalId != null) {
+ flushNewPartitions();
+ }
+ }
+
+ public Properties getKafkaProducerConfig() {
+ return kafkaProducerConfig;
+ }
+
+ public short getEpoch() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (short) getField(producerIdAndEpoch, "epoch");
+ }
+
+ public long getProducerId() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (long) getField(producerIdAndEpoch, "producerId");
+ }
+
+ /**
+ * Besides committing {@link org.apache.kafka.clients.producer.KafkaProducer#commitTransaction}
+ * is also adding new partitions to the transaction. flushNewPartitions method is moving this
+ * logic to pre-commit/flush, to make resumeTransaction simpler. Otherwise resumeTransaction
+ * would require to restore state of the not yet added/"in-flight" partitions.
+ */
+ private void flushNewPartitions() {
+ LOG.info("Flushing new partitions");
+ TransactionalRequestResult result = enqueueNewPartitions();
+ Object sender = getField("sender");
+ invoke(sender, "wakeup");
+ result.await();
+ }
+
+ /**
+ * Enqueues new transactions at the transaction manager and returns a {@link
+ * TransactionalRequestResult} that allows waiting on them.
+ *
+ * <p>If there are no new transactions we return a {@link TransactionalRequestResult} that is
+ * already done.
+ */
+ private TransactionalRequestResult enqueueNewPartitions() {
+ Object transactionManager = getField("transactionManager");
+ synchronized (transactionManager) {
+ Object newPartitionsInTransaction =
+ getField(transactionManager, "newPartitionsInTransaction");
+ Object newPartitionsInTransactionIsEmpty =
+ invoke(newPartitionsInTransaction, "isEmpty");
+ TransactionalRequestResult result;
+ if (newPartitionsInTransactionIsEmpty instanceof Boolean
+ && !((Boolean) newPartitionsInTransactionIsEmpty)) {
+ Object txnRequestHandler =
+ invoke(transactionManager, "addPartitionsToTransactionHandler");
+ invoke(
+ transactionManager,
+ "enqueueRequest",
+ new Class[] {txnRequestHandler.getClass().getSuperclass()},
+ new Object[] {txnRequestHandler});
+ result =
+ (TransactionalRequestResult)
+ getField(
+ txnRequestHandler,
+ txnRequestHandler.getClass().getSuperclass(),
+ "result");
+ } else {
+ // we don't have an operation but this operation string is also used in
+ // addPartitionsToTransactionHandler.
+ result = new TransactionalRequestResult("AddPartitionsToTxn");
+ result.done();
+ }
+ return result;
+ }
+ }
+
+ private static Object invoke(Object object, String methodName, Object... args) {
+ Class<?>[] argTypes = new Class[args.length];
+ for (int i = 0; i < args.length; i++) {
+ argTypes[i] = args[i].getClass();
+ }
+ return invoke(object, methodName, argTypes, args);
+ }
+
+ private static Object invoke(
+ Object object, String methodName, Class<?>[] argTypes, Object[] args) {
+ try {
+ Method method = object.getClass().getDeclaredMethod(methodName, argTypes);
+ method.setAccessible(true);
+ return method.invoke(object, args);
+ } catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ private Object getField(String fieldName) {
+ return getField(this, KafkaProducer.class, fieldName);
+ }
+
+ /**
+ * Gets and returns the field {@code fieldName} from the given Object {@code object} using
+ * reflection.
+ */
+ private static Object getField(Object object, String fieldName) {
+ return getField(object, object.getClass(), fieldName);
+ }
+
+ /**
+ * Gets and returns the field {@code fieldName} from the given Object {@code object} using
+ * reflection.
+ */
+ private static Object getField(Object object, Class<?> clazz, String fieldName) {
+ try {
+ Field field = clazz.getDeclaredField(fieldName);
+ field.setAccessible(true);
+ return field.get(object);
+ } catch (NoSuchFieldException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ /**
+ * Instead of obtaining producerId and epoch from the transaction coordinator, re-use previously
+ * obtained ones, so that we can resume transaction after a restart. Implementation of this
+ * method is based on {@link KafkaProducer#initTransactions}.
+ * https://github.com/apache/kafka/commit/5d2422258cb975a137a42a4e08f03573c49a387e#diff-f4ef1afd8792cd2a2e9069cd7ddea630
+ */
+ public void resumeTransaction(long producerId, short epoch) {
+ Preconditions.checkState(
+ producerId >= 0 && epoch >= 0,
+ "Incorrect values for producerId %s and epoch %s",
+ producerId,
+ epoch);
+ LOG.info(
+ "Attempting to resume transaction {} with producerId {} and epoch {}",
+ transactionalId,
+ producerId,
+ epoch);
+
+ Object transactionManager = getField("transactionManager");
+ synchronized (transactionManager) {
+ Object topicPartitionBookkeeper =
+ getField(transactionManager, "topicPartitionBookkeeper");
+
+ invoke(
+ transactionManager,
+ "transitionTo",
+ createTransactionManagerState("INITIALIZING"));
+ invoke(topicPartitionBookkeeper, "reset");
+
+ setField(
+ transactionManager,
+ "producerIdAndEpoch",
+ createProducerIdAndEpoch(producerId, epoch));
+
+ invoke(transactionManager, "transitionTo", createTransactionManagerState("READY"));
+
+ invoke(
+ transactionManager,
+ "transitionTo",
+ createTransactionManagerState("IN_TRANSACTION"));
+ setField(transactionManager, "transactionStarted", true);
+ }
+ }
+
+ private static Object createProducerIdAndEpoch(long producerId, short epoch) {
+ try {
+ Class<?> clazz =
+ Class.forName("org.apache.kafka.clients.producer.internals.ProducerIdAndEpoch");
+ Constructor<?> constructor = clazz.getDeclaredConstructors()[0];
+ constructor.setAccessible(true);
+ return constructor.newInstance(producerId, epoch);
+ } catch (ClassNotFoundException
+ | InvocationTargetException
+ | InstantiationException
+ | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ /**
+ * Sets the field {@code fieldName} on the given Object {@code object} to {@code value} using
+ * reflection.
+ */
+ private static void setField(Object object, String fieldName, Object value) {
+ try {
+ Field field = object.getClass().getDeclaredField(fieldName);
+ field.setAccessible(true);
+ field.set(object, value);
+ } catch (NoSuchFieldException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ private static Enum<?> createTransactionManagerState(String enumName) {
Review comment:
```suggestion
private static Enum<?> getTransactionManagerState(String enumName) {
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} In this mode the KafkaSink will write all messages in
+ * a Kafka transaction that will be committed to Kafka on a checkpoint. It is very important to
+ * ensure that you use unique {@link #transactionalIdPrefix}s across your applications running
+ * on the same Kafka cluster otherwise I may happen that multiple running jobs corrupt their
+ * transactions!
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
Review comment:
I'd skip it here. That's what the builder is for.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} In this mode the KafkaSink will write all messages in
+ * a Kafka transaction that will be committed to Kafka on a checkpoint. It is very important to
+ * ensure that you use unique {@link #transactionalIdPrefix}s across your applications running
+ * on the same Kafka cluster otherwise I may happen that multiple running jobs corrupt their
+ * transactions!
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
+ *
+ * @param <IN> type of the records written to Kafka
+ * @see KafkaSinkBuilder on how construct a KafkaSink
Review comment:
```suggestion
* @see KafkaSinkBuilder on how to construct a KafkaSink
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
+ * #setDeliverGuarantee(DeliveryGuarantee)} but keep in mind when using {@link
+ * DeliveryGuarantee#EXACTLY_ONCE} one must set the transactionalIdPrefix {@link
+ * #setTransactionalIdPrefix(String)}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+public class KafkaSinkBuilder<IN> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkBuilder.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private String transactionalIdPrefix = "kafka-sink";
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+ private String bootstrapServers;
+
+ /**
+ * Sets the wanted the {@link DeliveryGuarantee}. The default delivery guarantee is {@link
+ * #deliveryGuarantee}.
+ *
+ * @param deliveryGuarantee
+ * @return {@link KafkaSinkBuilder}
+ */
+ KafkaSinkBuilder<IN> setDeliverGuarantee(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "semantic");
+ return this;
+ }
+
+ /**
+ * Sets the configuration which used to instantiate all used {@link
+ * org.apache.kafka.clients.producer.KafkaProducer}.
+ *
+ * @param kafkaProducerConfig
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setKafkaProducerConfig(Properties kafkaProducerConfig) {
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ // set the producer configuration properties for kafka record key value serializers.
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG)) {
+ final long timeout = DEFAULT_KAFKA_TRANSACTION_TIMEOUT.toMillis();
+ checkState(
+ timeout < Integer.MAX_VALUE && timeout > 0,
+ "timeout does not fit into 32 bit integer");
+ kafkaProducerConfig.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, (int) timeout);
+ LOG.warn(
+ "Property [{}] not specified. Setting it to {}",
+ ProducerConfig.TRANSACTION_TIMEOUT_CONFIG,
+ DEFAULT_KAFKA_TRANSACTION_TIMEOUT);
+ }
+ return this;
+ }
+
+ /**
+ * Sets the {@link KafkaRecordSerializationSchema} that transforms incoming records to {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}s.
+ *
+ * @param recordSerializer
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setRecordSerializer(
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ ClosureCleaner.clean(
+ this.recordSerializer, ExecutionConfig.ClosureCleanerLevel.RECURSIVE, true);
+ return this;
+ }
+
+ /**
+ * Sets the prefix for all created transactionalIds if {@link DeliveryGuarantee#EXACTLY_ONCE} is
+ * configured.
+ *
+ * <p>It is mandatory to always set this value with {@link DeliveryGuarantee#EXACTLY_ONCE} to
+ * prevent corrupted transactions if multiple jobs using the KafkaSink run against the same
+ * Kafka Cluster. The default prefix is {@link #transactionalIdPrefix}.
+ *
+ * @param transactionalIdPrefix
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setTransactionalIdPrefix(String transactionalIdPrefix) {
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ return this;
+ }
+
+ /**
+ * Sets the Kafka bootstrap servers.
+ *
+ * @param bootstrapServers a comma separated list of valid URIs to reach the Kafka broker
+ * @return {@link KafkaSinkBuilder}
+ */
+ public KafkaSinkBuilder<IN> setBootstrapServers(String bootstrapServers) {
+ this.bootstrapServers = checkNotNull(bootstrapServers);
+ return this;
+ }
+
+ /**
+ * Constructs the {@link KafkaSink} with the configured properties.
+ *
+ * @return {@link KafkaSink}
+ */
+ public KafkaSink<IN> build() {
+ checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ checkNotNull(bootstrapServers);
+ if (deliveryGuarantee == DeliveryGuarantee.EXACTLY_ONCE) {
+ checkNotNull(
+ transactionalIdPrefix,
+ "EXACTLY_ONCE delivery guarantee requires a transactionIdPrefix to be set to provide unique transaction names across multiple KafkaSinks writing to the same Kafka cluster.");
+ }
Review comment:
nit: `checkNotNull` is typically used for parameters here `checkState` would be more appropriate. (KafkaSource is doing the same mistake)
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
Review comment:
Add a link to the KafkaSink description of guarantees.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
Review comment:
How about passing the combined list as parameter and then extract main as the first element?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkBuilder.java
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * <p>The following example shows the minimum setup to create a KafkaSink that writes String values
+ * to a Kafka topic.
+ *
+ * <pre>{@code
+ * KafkaSink<String> sink = KafkaSink
+ * .<String>builder
+ * .setBootstrapServers(MY_BOOTSTRAP_SERVERS)
+ * .setRecordSerializer(MY_RECORD_SERIALIZER)
+ * .setKafkaProducerConfig(MY_PRODUCER_CONFIG)
+ * .build();
+ * }</pre>
+ *
+ * <p>One can also configure different {@link DeliveryGuarantee} by using {@link
+ * #setDeliverGuarantee(DeliveryGuarantee)} but keep in mind when using {@link
+ * DeliveryGuarantee#EXACTLY_ONCE} one must set the transactionalIdPrefix {@link
+ * #setTransactionalIdPrefix(String)}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+public class KafkaSinkBuilder<IN> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkBuilder.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private String transactionalIdPrefix = "kafka-sink";
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+ private String bootstrapServers;
+
+ /**
+ * Sets the wanted the {@link DeliveryGuarantee}. The default delivery guarantee is {@link
+ * #deliveryGuarantee}.
+ *
+ * @param deliveryGuarantee
+ * @return {@link KafkaSinkBuilder}
+ */
+ KafkaSinkBuilder<IN> setDeliverGuarantee(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "semantic");
Review comment:
```suggestion
this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
```
or leave that param out as discussed.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
Review comment:
id seems to be UTF-8 encoded
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
Review comment:
no copy :p
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
+
+ private static byte readTransactionState(ByteBuffer buffer) {
+ // producerId
+ buffer.getLong();
+ // epoch
+ buffer.getShort();
+ // transactionTimeout
+ buffer.getInt();
+ // statusKey
+ return buffer.get();
+ }
+
+ enum TransactionState {
+ Empty(Byte.parseByte("0")),
+ Ongoing(Byte.parseByte("1")),
+ PrepareCommit(Byte.parseByte("2")),
+ PrepareAbort(Byte.parseByte("3")),
+ CompleteCommit(Byte.parseByte("4")),
+ CompleteAbort(Byte.parseByte("5")),
+ Dead(Byte.parseByte("6")),
+ PrepareEpochFence(Byte.parseByte("7"));
+
+ private final byte state;
+
+ TransactionState(byte state) {
+ this.state = state;
+ }
+
+ static TransactionState fromByte(byte state) {
+ return Arrays.stream(TransactionState.values())
+ .filter((value) -> value.state == state)
+ .findFirst()
+ .orElseThrow(
+ () ->
+ new IllegalArgumentException(
+ String.format(
+ "The given state %s is not supported.",
+ state)));
+ }
Review comment:
```suggestion
private static final Map<Byte, TransactionState> BYTE_TO_STATE =
Arrays.stream(TransactionState.values())
.collect(Collectors.toMap(e -> e.state, e -> e));
static TransactionState fromByte(byte state) {
TransactionState transactionState = BYTE_TO_STATE.get(state);
if (transactionState == null) {
throw new IllegalArgumentException(
String.format("The given state %s is not supported.", state));
}
return transactionState;
}
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
+
+ private static byte readTransactionState(ByteBuffer buffer) {
+ // producerId
+ buffer.getLong();
+ // epoch
+ buffer.getShort();
+ // transactionTimeout
+ buffer.getInt();
+ // statusKey
+ return buffer.get();
+ }
+
+ enum TransactionState {
+ Empty(Byte.parseByte("0")),
Review comment:
Welcome to Java! `Empty((byte) 0)`
Alternatively just have the ctor
```
TransactionState(int state) {
this.state = (byte) state;
}
```
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
Review comment:
Parsing should happen after prefix check.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
Review comment:
This is quite whacky:
- It may read too few records if Kafka broker has a small hiccup (1s)
- It most certainly reads too much if other applications append new transactions ever 1s (I'm assuming that this is not unusual for larger setups)
You should fetch the current committed offset before entering the loop and only break the loop iff the offset is reached.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
Review comment:
You should have a list of states in case of downscale (first entry should be the `main` or `active` entry).
Currently, we abort all transactions that are beyond this state, so if a restart happens before we committed everything, we may actually abort the transaction because we only store the `main` state in a new checkpoint. However, that would also not work with our enumeration of unknown task approach.
We probably need to encode the attempt after all ;) Can you create a follow-up ticket? The current limitation is that we assume that all transactions can be committed in-between downscale and the next checkpoint, which is okayish for a start.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/DefaultKafkaSinkContext.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Sink;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.common.PartitionInfo;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Context providing information to assist constructing a {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}.
+ */
+class DefaultKafkaSinkContext implements KafkaRecordSerializationSchema.KafkaSinkContext {
+
+ private final Sink.InitContext context;
Review comment:
Using the `InitContext` here makes it hard to create for testing. I'd pass the parallelism things directly.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
Review comment:
Please extract that logic into a test method and add unit tests + comments.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
+ recordSerializer.serialize(element, kafkaSinkContext, context.timestamp());
+ pendingRecords.incrementAndGet();
+ currentProducer.send(record, deliveryCallback);
+ }
+
+ @Override
+ public List<KafkaCommittable> prepareCommit(boolean flush) throws IOException {
+ flushRecords(flush);
+ if (!flush) {
+ currentProducer = beginTransaction();
+ }
+ final List<KafkaCommittable> committables = commit();
+ producers.add(currentProducer);
+ return committables;
+ }
+
+ @Override
+ public List<KafkaWriterState> snapshotState() throws IOException {
+ return ImmutableList.of(kafkaWriterState);
+ }
+
+ @Override
+ public void close() throws Exception {
+ currentProducer.close(Duration.ZERO);
Review comment:
producers for each?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
Review comment:
At this point we should immediately filter if the prefix matches.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
Review comment:
We should avoid `record` ;)
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
Review comment:
We should now enqueue a mail to avoid the `AtomicLong` and `volatile`.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
Review comment:
I'd index combined by subtask id for a non-scanning lookup.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
Review comment:
I'd also add a link in the first sentence such that it appears in all javadocs.
##########
File path: flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/DeliveryGuarantee.java
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base;
+
+/**
+ * Semantics that can be chosen.
+ * <li>{@link #EXACTLY_ONCE}
+ * <li>{@link #AT_LEAST_ONCE}
+ * <li>{@link #NONE}
+ */
+public enum DeliveryGuarantee {
+ EXACTLY_ONCE,
Review comment:
Thanks for expanding the javadoc. I was wondering if it wouldn't be better to move the javadoc for the individual constants to the constants? ;)
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriterState.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class KafkaWriterState {
+ private final int subtaskId;
+ private final long transactionalIdOffset;
+ private final String transactionalIdPrefix;
Review comment:
By storing it here, do you effectively allow users to change the prefix even when resuming from checkpoint?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
+ recordSerializer.serialize(element, kafkaSinkContext, context.timestamp());
+ pendingRecords.incrementAndGet();
+ currentProducer.send(record, deliveryCallback);
+ }
+
+ @Override
+ public List<KafkaCommittable> prepareCommit(boolean flush) throws IOException {
+ flushRecords(flush);
+ if (!flush) {
+ currentProducer = beginTransaction();
+ }
+ final List<KafkaCommittable> committables = commit();
+ producers.add(currentProducer);
+ return committables;
+ }
+
+ @Override
+ public List<KafkaWriterState> snapshotState() throws IOException {
+ return ImmutableList.of(kafkaWriterState);
+ }
+
+ @Override
+ public void close() throws Exception {
+ currentProducer.close(Duration.ZERO);
+ }
+
+ private KafkaWriterState recoverAndInitializeState(List<KafkaWriterState> recoveredStates) {
+ final int subtaskId = kafkaSinkContext.getParallelInstanceId();
+ if (recoveredStates.isEmpty()) {
+ final KafkaWriterState state =
+ new KafkaWriterState(transactionalIdPrefix, subtaskId, 0);
+ abortTransactions(getTransactionsToAbort(state, new ArrayList<>()));
+ return state;
+ }
+ final Map<Integer, KafkaWriterState> taskOffsetMapping =
+ recoveredStates.stream()
+ .collect(
+ Collectors.toMap(
+ KafkaWriterState::getSubtaskId, Function.identity()));
+ if (!taskOffsetMapping.containsKey(subtaskId)) {
+ throw new IllegalStateException(
+ "It is expected that state from previous executions is distributed to the same subtask id.");
+ }
+ final KafkaWriterState lastState = taskOffsetMapping.get(subtaskId);
+ taskOffsetMapping.remove(subtaskId);
+ abortTransactions(
+ getTransactionsToAbort(lastState, new ArrayList<>(taskOffsetMapping.values())));
+ if (!lastState.getTransactionalIdPrefix().equals(transactionalIdPrefix)) {
+ LOG.warn(
+ "Transactional id prefix from previous execution {} has changed to {}.",
+ lastState.getTransactionalIdPrefix(),
+ transactionalIdPrefix);
+ return new KafkaWriterState(transactionalIdPrefix, subtaskId, 0);
+ }
+ return new KafkaWriterState(
+ transactionalIdPrefix, subtaskId, lastState.getTransactionalIdOffset());
+ }
+
+ private void abortTransactions(Set<String> transactionsToAbort) {
+ transactionsToAbort.forEach(
+ transaction -> {
+ // don't mess with the original configuration or any other
+ // properties of the
+ // original object
+ // -> create an internal kafka producer on our own and do not rely
+ // on
+ // initTransactionalProducer().
+ final Properties myConfig = new Properties();
+ myConfig.putAll(kafkaProducerConfig);
+ myConfig.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transaction);
+ LOG.info("Aborting Kafka transaction {}.", transaction);
+ FlinkKafkaInternalProducer<byte[], byte[]> kafkaProducer = null;
Review comment:
We could use try-with-resource if we relay `close` to `close(0)` in `FlinkKafkaInternalProducer`.
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkITCase.java
##########
@@ -0,0 +1,634 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.StateBackendOptions;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.test.util.TestUtils;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.testutils.junit.SharedReference;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.CreateTopicsResult;
+import org.apache.kafka.clients.admin.DeleteTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.LongStream;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+/** Tests for using KafkaSink writing to a Kafka cluster. */
+public class KafkaSinkITCase extends TestLogger {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkITCase.class);
+ private static final Slf4jLogConsumer LOG_CONSUMER = new Slf4jLogConsumer(LOG);
+ private static final String INTER_CONTAINER_KAFKA_ALIAS = "kafka";
+ private static final Network NETWORK = Network.newNetwork();
+ private static final int ZK_TIMEOUT_MILLIS = 30000;
+ private static final short TOPIC_REPLICATION_FACTOR = 1;
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+
+ private String topic;
+ private SharedReference<AtomicLong> emittedRecordsCount;
+ private SharedReference<AtomicLong> emittedRecordsWithCheckpoint;
+ private SharedReference<AtomicBoolean> failed;
+ private SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ @ClassRule
+ public static final KafkaContainer KAFKA_CONTAINER =
+ new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.5.2"))
+ .withEmbeddedZookeeper()
+ .withEnv(
+ ImmutableMap.of(
+ "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR",
+ "1",
+ "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
+ String.valueOf(Duration.ofHours(2).toMillis()),
+ "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR",
+ "1",
+ "KAFKA_MIN_INSYNC_REPLICAS",
+ "1"))
+ .withNetwork(NETWORK)
+ .withLogConsumer(LOG_CONSUMER)
+ .withNetworkAliases(INTER_CONTAINER_KAFKA_ALIAS);
+
+ @Rule public final SharedObjects sharedObjects = SharedObjects.create();
+
+ @Rule public final TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void setUp() throws ExecutionException, InterruptedException, TimeoutException {
+ emittedRecordsCount = sharedObjects.add(new AtomicLong());
+ emittedRecordsWithCheckpoint = sharedObjects.add(new AtomicLong());
+ failed = sharedObjects.add(new AtomicBoolean(false));
+ lastCheckpointedRecord = sharedObjects.add(new AtomicLong(0));
+ topic = UUID.randomUUID().toString();
+ createTestTopic(topic, 1, TOPIC_REPLICATION_FACTOR);
+ }
+
+ @After
+ public void tearDown() throws ExecutionException, InterruptedException, TimeoutException {
+ deleteTestTopic(topic);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithAtLeastOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.AT_LEAST_ONCE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithNoneGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.NONE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithExactlyOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.EXACTLY_ONCE, emittedRecordsWithCheckpoint);
+ }
+
+ @Test
+ public void testRecoveryWithAtLeastOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.AT_LEAST_ONCE,
+ (records) ->
+ assertThat(records, hasItems(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L)));
+ }
+
+ @Test
+ public void testRecoveryWithExactlyOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.EXACTLY_ONCE,
+ (records) ->
+ assertEquals(
+ records,
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList())));
+ }
+
+ @Test
+ public void testAbortTransactionsOfPendingCheckpointsAfterFailure() throws Exception {
+ // Run a first job failing during the async phase of a checkpoint to leave some
+ // lingering transactions
+ final Configuration config = new Configuration();
+ config.setString(StateBackendOptions.STATE_BACKEND, "filesystem");
+ final File checkpointDir = temp.newFolder();
+ config.setString(
+ CheckpointingOptions.CHECKPOINTS_DIRECTORY, checkpointDir.toURI().toString());
+ config.set(
+ ExecutionCheckpointingOptions.EXTERNALIZED_CHECKPOINT,
+ CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
+ config.set(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 2);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(1), config, "firstPrefix");
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ final File completedCheckpoint = TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
+
+ config.set(SavepointConfigOptions.SAVEPOINT_PATH, completedCheckpoint.toURI().toString());
+
+ // Run a second job which aborts all lingering transactions and new consumer should
+ // immediately see the newly written records
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, "newPrefix");
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ @Test
+ public void testAbortTransactionsAfterScaleInBeforeFirstCheckpoint() throws Exception {
+ // Run a first job opening 5 transactions one per subtask and fail in async checkpoint phase
+ final Configuration config = new Configuration();
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 5);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(0), config, null);
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ assertTrue(deserializeValues(drainAllRecordsFromTopic(topic)).isEmpty());
+
+ // Second job aborts all transactions from previous runs with higher parallelism
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 1);
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, null);
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ private void executeWithMapper(
+ MapFunction<Long, Long> mapper,
+ Configuration config,
+ @Nullable String transactionalIdPrefix)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment(config);
+ env.enableCheckpointing(100L);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ final DataStreamSource<Long> source = env.fromSequence(1, 10);
+ final DataStream<Long> stream = source.map(mapper);
+ final KafkaSinkBuilder<Long> builder =
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic));
+ if (transactionalIdPrefix == null) {
+ transactionalIdPrefix = "kafka-sink";
+ }
+ builder.setTransactionalIdPrefix(transactionalIdPrefix);
+ stream.sinkTo(builder.build());
+ env.execute();
+ checkProducerLeak();
+ }
+
+ private void testRecoveryWithAssertion(
+ DeliveryGuarantee guarantee, java.util.function.Consumer<List<Long>> recordsAssertion)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment();
+ env.enableCheckpointing(300L);
+ DataStreamSource<Long> source = env.fromSequence(1, 10);
+ DataStream<Long> stream =
+ source.map(new FailingCheckpointMapper(failed, lastCheckpointedRecord));
+
+ stream.sinkTo(
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(guarantee)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic))
+ .setTransactionalIdPrefix("kafka-sink")
+ .build());
+ env.execute();
+
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ recordsAssertion.accept(deserializeValues(collectedRecords));
+ checkProducerLeak();
+ }
+
+ private void writeRecordsToKafka(
+ DeliveryGuarantee deliveryGuarantee, SharedReference<AtomicLong> expectedRecords)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment();
+ env.enableCheckpointing(1000L);
+ final DataStream<Long> source =
+ env.addSource(
+ new InfiniteIntegerSource(
+ emittedRecordsCount, emittedRecordsWithCheckpoint));
+ source.sinkTo(
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setDeliverGuarantee(deliveryGuarantee)
+ .setRecordSerializer(new RecordSerializer(topic))
+ .setTransactionalIdPrefix("kafka-sink")
+ .build());
+ env.execute();
+
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ final long recordsCount = expectedRecords.get().get();
+ assertEquals(collectedRecords.size(), recordsCount);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(0, recordsCount).boxed().collect(Collectors.toList()));
+ checkProducerLeak();
+ }
+
+ private static List<Long> deserializeValues(List<ConsumerRecord<byte[], byte[]>> records) {
+ return records.stream()
+ .map(
+ record -> {
+ final ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ final byte[] value = record.value();
+ buffer.put(value, 0, value.length);
+ buffer.flip();
+ return buffer.getLong();
+ })
+ .collect(Collectors.toList());
+ }
+
+ private static Properties getKafkaClientConfiguration() {
+ final Properties standardProps = new Properties();
+ standardProps.put("bootstrap.servers", KAFKA_CONTAINER.getBootstrapServers());
+ standardProps.put("group.id", UUID.randomUUID().toString());
+ standardProps.put("enable.auto.commit", false);
+ standardProps.put("auto.offset.reset", "earliest");
+ standardProps.put("max.partition.fetch.bytes", 256);
+ standardProps.put("zookeeper.session.timeout.ms", ZK_TIMEOUT_MILLIS);
+ standardProps.put("zookeeper.connection.timeout.ms", ZK_TIMEOUT_MILLIS);
+ return standardProps;
+ }
+
+ private Consumer<byte[], byte[]> createTestConsumer(String topic) {
+ final Properties consumerConfig = new Properties();
+ consumerConfig.putAll(getKafkaClientConfiguration());
+ consumerConfig.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ consumerConfig.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ consumerConfig.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ final KafkaConsumer<byte[], byte[]> kafkaConsumer = new KafkaConsumer<>(consumerConfig);
+ kafkaConsumer.subscribe(Collections.singletonList(topic));
+ return kafkaConsumer;
+ }
+
+ private void createTestTopic(String topic, int numPartitions, short replicationFactor)
+ throws ExecutionException, InterruptedException, TimeoutException {
+ Map<String, Object> properties = new HashMap<>();
+ properties.put(
+ CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
+ KAFKA_CONTAINER.getBootstrapServers());
+ try (AdminClient admin = AdminClient.create(properties)) {
+ final CreateTopicsResult result =
+ admin.createTopics(
+ Collections.singletonList(
+ new NewTopic(topic, numPartitions, replicationFactor)));
+ result.all().get(1, TimeUnit.MINUTES);
+ }
+ }
+
+ private void deleteTestTopic(String topic)
+ throws ExecutionException, InterruptedException, TimeoutException {
+ Map<String, Object> properties = new HashMap<>();
+ properties.put(
+ CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
+ KAFKA_CONTAINER.getBootstrapServers());
+ try (AdminClient admin = AdminClient.create(properties)) {
+ final DeleteTopicsResult result = admin.deleteTopics(Collections.singletonList(topic));
+ result.all().get(1, TimeUnit.MINUTES);
+ }
+ }
+
+ private List<ConsumerRecord<byte[], byte[]>> drainAllRecordsFromTopic(String topic) {
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords = new ArrayList<>();
+ try (Consumer<byte[], byte[]> consumer = createTestConsumer(topic)) {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ // Drain the kafka topic till all records are consumed
+ while (!records.isEmpty()) {
+ records.records(topic).forEach(collectedRecords::add);
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ }
+ return collectedRecords;
+ }
+
+ private static class RecordSerializer implements KafkaRecordSerializationSchema<Long> {
+
+ private final String topic;
+
+ public RecordSerializer(String topic) {
+ this.topic = topic;
+ }
+
+ @Override
+ public ProducerRecord<byte[], byte[]> serialize(
+ Long element, KafkaSinkContext context, Long timestamp) {
+ final ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ buffer.putLong(element);
+ return new ProducerRecord<>(topic, 0, null, null, buffer.array());
+ }
+ }
+
+ private static class FailAsyncCheckpointMapper
+ implements MapFunction<Long, Long>, CheckpointedFunction {
+ private static final ListStateDescriptor<Integer> stateDescriptor =
+ new ListStateDescriptor<>("test-state", new SlowSerializer());
+ private int failAfterCheckpoint;
+
+ private ListState<Integer> state;
+
+ public FailAsyncCheckpointMapper(int failAfterCheckpoint) {
+ this.failAfterCheckpoint = failAfterCheckpoint;
+ }
+
+ @Override
+ public Long map(Long value) throws Exception {
+ Thread.sleep(100);
+ return value;
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ state.clear();
+ if (failAfterCheckpoint <= 0) {
+ // Trigger a failure in the serializer
+ state.add(-1);
+ } else {
+ state.add(1);
+ }
+ failAfterCheckpoint--;
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ state = context.getOperatorStateStore().getListState(stateDescriptor);
+ }
+ }
+
+ private static class SlowSerializer extends TypeSerializerSingleton<Integer> {
+
+ @Override
+ public boolean isImmutableType() {
+ return false;
+ }
+
+ @Override
+ public Integer createInstance() {
+ return 1;
+ }
+
+ @Override
+ public Integer copy(Integer from) {
+ return from;
+ }
+
+ @Override
+ public Integer copy(Integer from, Integer reuse) {
+ return from;
+ }
+
+ @Override
+ public int getLength() {
+ return 0;
+ }
+
+ @Override
+ public void serialize(Integer record, DataOutputView target) throws IOException {
+ if (record != -1) {
+ return;
+ }
+ throw new RuntimeException("Expected failure during async checkpoint phase");
+ }
+
+ @Override
+ public Integer deserialize(DataInputView source) throws IOException {
+ return 1;
+ }
+
+ @Override
+ public Integer deserialize(Integer reuse, DataInputView source) throws IOException {
+ return 1;
+ }
+
+ @Override
+ public void copy(DataInputView source, DataOutputView target) throws IOException {}
+
+ @Override
+ public TypeSerializerSnapshot<Integer> snapshotConfiguration() {
+ return new SlowSerializerSnapshot();
+ }
+ }
+
+ /** Snapshot used in {@link FailAsyncCheckpointMapper}. */
+ public static class SlowSerializerSnapshot extends SimpleTypeSerializerSnapshot<Integer> {
+ public SlowSerializerSnapshot() {
+ super(SlowSerializer::new);
+ }
+ }
+
+ /** Fails after a checkpoint is taken and the next record was emitted. */
+ private static class FailingCheckpointMapper
+ implements MapFunction<Long, Long>, CheckpointListener, CheckpointedFunction {
+
+ private final SharedReference<AtomicBoolean> failed;
+ private final SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ private volatile long lastSeenRecord;
+ private volatile long checkpointedRecord;
+ private volatile long lastCheckpointId = 0;
+ private final AtomicInteger emittedBetweenCheckpoint = new AtomicInteger(0);
+
+ FailingCheckpointMapper(
+ SharedReference<AtomicBoolean> failed,
+ SharedReference<AtomicLong> lastCheckpointedRecord) {
+ this.failed = failed;
+ this.lastCheckpointedRecord = lastCheckpointedRecord;
+ }
+
+ @Override
+ public Long map(Long value) throws Exception {
+ lastSeenRecord = value;
+ if (lastCheckpointId >= 1
+ && emittedBetweenCheckpoint.get() > 0
+ && !failed.get().get()) {
+ failed.get().set(true);
+ throw new RuntimeException("Planned exception.");
+ }
+ Thread.sleep(50);
+ emittedBetweenCheckpoint.incrementAndGet();
+ return value;
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ lastCheckpointId = checkpointId;
+ emittedBetweenCheckpoint.set(0);
+ lastCheckpointedRecord.get().set(checkpointedRecord);
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ checkpointedRecord = lastSeenRecord;
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {}
+ }
+
+ private void checkProducerLeak() {
+ for (Thread t : Thread.getAllStackTraces().keySet()) {
+ if (t.getName().contains("kafka-producer-network-thread")) {
+ fail("Detected producer leak. Thread name: " + t.getName());
+ }
+ }
+ }
+
+ /**
+ * Exposes information about how man records have been emitted overall and at the beginning of a
+ * checkpoint.
+ */
+ private static final class InfiniteIntegerSource
Review comment:
Can you just use a `env.fromSequence(0, Long.MAX_VALUE)` with a chained map that implements this functionality? You are making our future lives harder :/
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriterITCase.java
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import java.time.Duration;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.contains;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for the standalone KafkaWriter. */
+@RunWith(Parameterized.class)
+public class KafkaWriterITCase {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriterITCase.class);
+ private static final Slf4jLogConsumer LOG_CONSUMER = new Slf4jLogConsumer(LOG);
+ private static final String INTER_CONTAINER_KAFKA_ALIAS = "kafka";
+ private static final Network NETWORK = Network.newNetwork();
+
+ private final DeliveryGuarantee guarantee;
+
+ private MockMetricGroup metricGroup;
+
+ @ClassRule
+ public static final KafkaContainer KAFKA_CONTAINER =
+ new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.5.2"))
+ .withEmbeddedZookeeper()
+ .withEnv(
+ ImmutableMap.of(
+ "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR",
+ "1",
+ "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
+ String.valueOf(Duration.ofHours(2).toMillis()),
+ "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR",
+ "1",
+ "KAFKA_MIN_INSYNC_REPLICAS",
+ "1"))
+ .withNetwork(NETWORK)
+ .withLogConsumer(LOG_CONSUMER)
+ .withNetworkAliases(INTER_CONTAINER_KAFKA_ALIAS);
+
+ @Before
+ public void setUp() {
+ metricGroup = new MockMetricGroup();
+ }
+
+ @Parameterized.Parameters
+ public static List<DeliveryGuarantee> guarantees() {
+ return ImmutableList.of(
+ DeliveryGuarantee.NONE,
+ DeliveryGuarantee.AT_LEAST_ONCE,
+ DeliveryGuarantee.EXACTLY_ONCE);
+ }
+
+ public KafkaWriterITCase(DeliveryGuarantee guarantee) {
+ this.guarantee = guarantee;
+ }
+
+ @Test
+ public void testRegisterMetrics() throws Exception {
+ try (final KafkaWriter<Integer> ignored =
+ createWriterWithConfiguration(getKafkaClientConfiguration())) {
+ assertThat(metricGroup.addedGroups, contains("KafkaProducer"));
+ assertFalse(metricGroup.registeredGauges.isEmpty());
+ }
+ }
+
+ @Test
+ public void testNotRegisterMetrics() throws Exception {
+ final Properties config = getKafkaClientConfiguration();
+ config.put("flink.disable-metrics", "true");
+ try (final KafkaWriter<Integer> ignored = createWriterWithConfiguration(config)) {
+ assertTrue(metricGroup.addedGroups.isEmpty());
+ assertTrue(metricGroup.registeredGauges.isEmpty());
+ }
+ }
+
+ private KafkaWriter<Integer> createWriterWithConfiguration(Properties config) {
+ return new KafkaWriter<>(
+ guarantee,
+ config,
+ "test-prefix",
+ new SinkInitContext(metricGroup),
+ new DummyRecordSerializer(),
+ new DummySchemaContext(),
+ ImmutableList.of());
+ }
+
+ private static Properties getKafkaClientConfiguration() {
+ final Properties standardProps = new Properties();
+ standardProps.put("bootstrap.servers", KAFKA_CONTAINER.getBootstrapServers());
+ standardProps.put("group.id", "kafkaWriter-tests");
+ standardProps.put("enable.auto.commit", false);
+ standardProps.put("key.serializer", ByteArraySerializer.class.getName());
+ standardProps.put("value.serializer", ByteArraySerializer.class.getName());
+ standardProps.put("auto.offset.reset", "earliest");
+ return standardProps;
+ }
+
+ private static class MockMetricGroup implements MetricGroup {
Review comment:
Can you check if we can use the `MetricListener`?
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
+ return record -> {
+ final ByteBuffer keyBuffer = ByteBuffer.wrap(record.key());
+ checkKafkaSchemaVersionMatches(keyBuffer);
+ // Ignore 2 bytes because Kafka's internal representation
+ keyBuffer.getShort();
+ final String transactionalId = StandardCharsets.US_ASCII.decode(keyBuffer).toString();
+
+ final ByteBuffer valueBuffer = ByteBuffer.wrap(record.value());
+ checkKafkaSchemaVersionMatches(valueBuffer);
+ final TransactionState state =
+ TransactionState.fromByte(readTransactionState(valueBuffer));
+
+ LOG.debug("Transaction {} is in state {}", transactionalId, state);
+
+ // If the transaction is in a final state ignore it
+ if (isTransactionInFinalState(state)) {
+ openTransactions.remove(transactionalId);
+ return;
+ }
+
+ final Optional<KafkaWriterState> openTransactionOpt =
+ fromTransactionalId(transactionalId);
+ // If the transactionalId does not follow the format ignore it
+ if (!openTransactionOpt.isPresent()) {
+ return;
+ }
+
+ final KafkaWriterState openTransaction = openTransactionOpt.get();
+ // If the transactionalId prefixes differ ignore
+ if (!openTransaction
+ .getTransactionalIdPrefix()
+ .equals(main.getTransactionalIdPrefix())) {
+ LOG.debug(
+ "The transactionalId prefixes differ. Open: {}, Recovered: {}",
+ openTransaction.getTransactionalIdPrefix(),
+ main.getTransactionalIdPrefix());
+ return;
+ }
+
+ final int openSubtaskIndex = openTransaction.getSubtaskId();
+ final long openCheckpointOffset = openTransaction.getTransactionalIdOffset();
+
+ final boolean commonRemainder =
+ openSubtaskIndex % numberOfParallelSubtasks == main.getSubtaskId()
+ && openSubtaskIndex != main.getSubtaskId();
+ final boolean sameSubtaskHigherCheckpoint =
+ hasSameSubtaskWithHigherCheckpoint(openSubtaskIndex, openCheckpointOffset);
+ final boolean higherSubtasksWithFirstCheckpoint =
+ minOthers.isPresent()
+ && minOthers.getAsInt() < openSubtaskIndex
+ && openCheckpointOffset == 0;
+
+ if (!commonRemainder
+ && !sameSubtaskHigherCheckpoint
+ && !higherSubtasksWithFirstCheckpoint) {
+ return;
+ }
+ openTransactions.add(transactionalId);
+ };
+ }
+
+ @Override
+ public void close() {
+ consumer.close();
+ }
+
+ private boolean hasSameSubtaskWithHigherCheckpoint(
+ int openSubtaskIndex, long openCheckpointOffset) {
+ return combined.stream()
+ .anyMatch(
+ state ->
+ state.getSubtaskId() == openSubtaskIndex
+ && state.getTransactionalIdOffset()
+ <= openCheckpointOffset);
+ }
+
+ private static boolean isTransactionInFinalState(TransactionState state) {
+ return TERMINAL_TRANSACTION_STATES.contains(state);
+ }
+
+ private static Optional<KafkaWriterState> fromTransactionalId(String transactionalId) {
+ final String[] splits = transactionalId.split(TRANSACTIONAL_ID_DELIMITER);
+ if (splits.length < 3) {
+ LOG.debug("Transaction {} was not created by the Flink Kafka sink", transactionalId);
+ return Optional.empty();
+ }
+ try {
+ final long checkpointOffset = Long.parseLong(splits[splits.length - 1]);
+ final int subtaskId = Integer.parseInt(splits[splits.length - 2]);
+ return Optional.of(
+ new KafkaWriterState(
+ String.join(
+ TRANSACTIONAL_ID_DELIMITER,
+ Arrays.copyOfRange(splits, 0, splits.length - 2)),
+ subtaskId,
+ checkpointOffset));
+ } catch (NumberFormatException e) {
+ LOG.debug(
+ "Transaction {} was not created by the Flink Kafka sink: {}",
+ transactionalId,
+ e);
+ return Optional.empty();
+ }
+ }
Review comment:
Please add unit test, especially to cover a prefix with the delimiter (code looks good) ;)
Also please avoid the costly `split` and use Guava's Splitter instead.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaCommitter.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Committer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.errors.InvalidTxnStateException;
+import org.apache.kafka.common.errors.ProducerFencedException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Committer implementation for {@link KafkaSink}
+ *
+ * <p>The committer is responsible to finalize the Kafka transactions by committing them.
+ */
+class KafkaCommitter implements Committer<KafkaCommittable> {
+
+ private final Properties kafkaProducerConfig;
+
+ KafkaCommitter(Properties kafkaProducerConfig) {
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ }
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaCommitter.class);
+
+ @Override
+ public List<KafkaCommittable> commit(List<KafkaCommittable> committables) throws IOException {
+ committables.forEach(this::commitTransaction);
+ return Collections.emptyList();
+ }
+
+ @Override
+ public void close() throws Exception {}
+
+ private void commitTransaction(KafkaCommittable committable) {
+ final String transactionalId = committable.getTransactionalId();
+ LOG.debug("Committing Kafka transaction {}", transactionalId);
+ try (FlinkKafkaInternalProducer<?, ?> producer =
+ new FlinkKafkaInternalProducer<>(createKafkaProducerConfig(transactionalId))) {
+ producer.resumeTransaction(committable.getProducerId(), committable.getEpoch());
+ producer.commitTransaction();
+ } catch (InvalidTxnStateException | ProducerFencedException e) {
+ // That means we have committed this transaction before.
+ LOG.warn(
+ "Encountered error {} while recovering transaction {}. "
+ + "Presumably this transaction has been already committed before",
+ e,
+ committable);
Review comment:
Upon re-reading, I'm assuming that it actually fails elsewhere and he just sees the warning over and over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 5916417a1266f4af34d387dd2e61a10673a1a9fe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535)
* 864808edf90294e0fa1386c321ae752a7e076b97 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25fb6f9a84ab137be93a04bfad41c4ab021a3b4b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464)
* 1e41696e78086dda1970f6e50e3125d0271c056b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494)
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 77d980b81bcd178befe0a1f3762234a2e5b79222 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25fb6f9a84ab137be93a04bfad41c4ab021a3b4b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464)
* 1e41696e78086dda1970f6e50e3125d0271c056b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 864808edf90294e0fa1386c321ae752a7e076b97 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639)
* e8c6ec06f2263f5704ccf597494c43a1bd0e5893 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r681534451
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/DefaultKafkaSinkContext.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.connector.sink.Sink;
+
+import org.apache.kafka.common.PartitionInfo;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+/**
+ * Context providing information to assist constructing a {@link
+ * org.apache.kafka.clients.producer.ProducerRecord}.
+ */
+class DefaultKafkaSinkContext implements KafkaRecordSerializationSchema.KafkaSinkContext {
+
+ private final Sink.InitContext context;
+ private final FlinkKafkaInternalProducer<?, ?> producer;
+
+ public DefaultKafkaSinkContext(
+ Sink.InitContext context, FlinkKafkaInternalProducer<?, ?> producer) {
+ this.context = context;
+ this.producer = producer;
+ }
+
+ @Override
+ public int getParallelInstanceId() {
+ return context.getSubtaskId();
+ }
+
+ @Override
+ public int getNumberOfParallelInstances() {
+ return context.getNumberOfParallelSubtasks();
+ }
+
+ @Override
+ public int[] getPartitionsForTopic(String topic) {
+ // the fetched list is immutable, so we're creating a mutable copy in order to sort it
+ final List<PartitionInfo> partitionsList = new ArrayList<>(producer.partitionsFor(topic));
+
+ return partitionsList.stream()
+ .sorted(Comparator.comparing(PartitionInfo::partition))
+ .map(PartitionInfo::partition)
+ .mapToInt(Integer::intValue)
+ .toArray();
Review comment:
Cache? If cache, please add documentation that it's cached.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/FlinkKafkaInternalProducer.java
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.TransactionalRequestResult;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.Properties;
+
+/**
+ * A {@link KafkaProducer} that exposes private fields to allow resume producing from a given state.
+ */
+class FlinkKafkaInternalProducer<K, V> extends KafkaProducer<K, V> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkKafkaInternalProducer.class);
+
+ @Nullable private final String transactionalId;
+
+ public FlinkKafkaInternalProducer(Properties properties) {
+ super(properties);
+ this.transactionalId = properties.getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
+ }
+
+ @Override
+ public Map<MetricName, ? extends Metric> metrics() {
+ return null;
+ }
+
+ @Override
+ public void close() {
+ throw new UnsupportedOperationException(
+ "Close without timeout is now allowed because it can leave lingering Kafka threads.");
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ if (transactionalId != null) {
+ flushNewPartitions();
+ }
+ }
+
+ public short getEpoch() {
+ Object transactionManager = getField(this, "transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (short) getField(producerIdAndEpoch, "epoch");
+ }
+
+ public long getProducerId() {
+ Object transactionManager = getField(this, "transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (long) getField(producerIdAndEpoch, "producerId");
+ }
+
+ /**
+ * Besides committing {@link org.apache.kafka.clients.producer.KafkaProducer#commitTransaction}
+ * is also adding new partitions to the transaction. flushNewPartitions method is moving this
+ * logic to pre-commit/flush, to make resumeTransaction simpler. Otherwise resumeTransaction
+ * would require to restore state of the not yet added/"in-flight" partitions.
+ */
+ private void flushNewPartitions() {
+ LOG.info("Flushing new partitions");
+ TransactionalRequestResult result = enqueueNewPartitions();
+ Object sender = getField(this, "sender");
+ invoke(sender, "wakeup");
+ result.await();
+ }
+
+ /**
+ * Enqueues new transactions at the transaction manager and returns a {@link
+ * TransactionalRequestResult} that allows waiting on them.
+ *
+ * <p>If there are no new transactions we return a {@link TransactionalRequestResult} that is
+ * already done.
+ */
+ private TransactionalRequestResult enqueueNewPartitions() {
+ Object transactionManager = getField(this, "transactionManager");
+ synchronized (transactionManager) {
+ Object newPartitionsInTransaction =
+ getField(transactionManager, "newPartitionsInTransaction");
+ Object newPartitionsInTransactionIsEmpty =
+ invoke(newPartitionsInTransaction, "isEmpty");
+ TransactionalRequestResult result;
+ if (newPartitionsInTransactionIsEmpty instanceof Boolean
+ && !((Boolean) newPartitionsInTransactionIsEmpty)) {
+ Object txnRequestHandler =
+ invoke(transactionManager, "addPartitionsToTransactionHandler");
+ invoke(
+ transactionManager,
+ "enqueueRequest",
+ new Class[] {txnRequestHandler.getClass().getSuperclass()},
+ new Object[] {txnRequestHandler});
+ result =
+ (TransactionalRequestResult)
+ getField(
+ txnRequestHandler,
+ txnRequestHandler.getClass().getSuperclass(),
+ "result");
+ } else {
+ // we don't have an operation but this operation string is also used in
+ // addPartitionsToTransactionHandler.
+ result = new TransactionalRequestResult("AddPartitionsToTxn");
+ result.done();
+ }
+ return result;
+ }
+ }
+
+ private static Object invoke(Object object, String methodName, Object... args) {
+ Class<?>[] argTypes = new Class[args.length];
+ for (int i = 0; i < args.length; i++) {
+ argTypes[i] = args[i].getClass();
+ }
+ return invoke(object, methodName, argTypes, args);
+ }
+
+ private static Object invoke(
+ Object object, String methodName, Class<?>[] argTypes, Object[] args) {
+ try {
+ Method method =
+ object.getClass().getSuperclass().getDeclaredMethod(methodName, argTypes);
+ method.setAccessible(true);
+ return method.invoke(object, args);
+ } catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ /**
+ * Gets and returns the field {@code fieldName} from the given Object {@code object} using
+ * reflection.
+ */
+ private static Object getField(Object object, String fieldName) {
+ return getField(object, object.getClass(), fieldName);
+ }
+
+ /**
+ * Gets and returns the field {@code fieldName} from the given Object {@code object} using
+ * reflection.
+ */
+ private static Object getField(Object object, Class<?> clazz, String fieldName) {
+ try {
+ Field field = clazz.getSuperclass().getDeclaredField(fieldName);
+ field.setAccessible(true);
+ return field.get(object);
+ } catch (NoSuchFieldException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ /**
+ * Instead of obtaining producerId and epoch from the transaction coordinator, re-use previously
+ * obtained ones, so that we can resume transaction after a restart. Implementation of this
+ * method is based on {@link KafkaProducer#initTransactions}.
+ * https://github.com/apache/kafka/commit/5d2422258cb975a137a42a4e08f03573c49a387e#diff-f4ef1afd8792cd2a2e9069cd7ddea630
+ */
+ public void resumeTransaction(long producerId, short epoch) {
+ Preconditions.checkState(
+ producerId >= 0 && epoch >= 0,
+ "Incorrect values for producerId %s and epoch %s",
+ producerId,
+ epoch);
+ LOG.info(
+ "Attempting to resume transaction {} with producerId {} and epoch {}",
+ transactionalId,
+ producerId,
+ epoch);
+
+ Object transactionManager = getField(this, "transactionManager");
+ synchronized (transactionManager) {
+ Object topicPartitionBookkeeper =
+ getField(transactionManager, "topicPartitionBookkeeper");
+
+ invoke(
+ transactionManager,
+ "transitionTo",
+ getEnum(
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State.INITIALIZING"));
+ invoke(topicPartitionBookkeeper, "reset");
+
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ setField(producerIdAndEpoch, "producerId", producerId);
+ setField(producerIdAndEpoch, "epoch", epoch);
+
+ invoke(
+ transactionManager,
+ "transitionTo",
+ getEnum(
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State.READY"));
+
+ invoke(
+ transactionManager,
+ "transitionTo",
+ getEnum(
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State.IN_TRANSACTION"));
+ setField(transactionManager, "transactionStarted", true);
+ }
+ }
+
+ /**
+ * Sets the field {@code fieldName} on the given Object {@code object} to {@code value} using
+ * reflection.
+ */
+ private static void setField(Object object, String fieldName, Object value) {
+ try {
+ Field field = object.getClass().getDeclaredField(fieldName);
+ field.setAccessible(true);
+ field.set(object, value);
+ } catch (NoSuchFieldException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
+
+ private static Enum<?> getEnum(String enumFullName) {
Review comment:
Please split arguments on call-site.
##########
File path: flink-connectors/flink-connector-kafka/pom.xml
##########
@@ -78,6 +78,18 @@ under the License.
<version>${kafka.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kafka_${scala.binary.version}</artifactId>
+ <version>${kafka.version}</version>
+ <exclusions>
Review comment:
Either exclude all or try to avoid this dependency.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaCommittable.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import java.util.Objects;
+import java.util.Properties;
+
+/**
+ * This class holds the necessary information to construct a new {@link FlinkKafkaInternalProducer}
+ * to commit transactions in {@link KafkaCommitter}.
+ */
+class KafkaCommittable {
+
+ private final long producerId;
+ private final short epoch;
+ private final Properties kafkaProducerConfig;
Review comment:
Replace with `transactionalId` and inject Properties in `createCommitter`.
##########
File path: flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/DeliveryGuarantee.java
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.base;
+
+/**
+ * Semantics that can be chosen.
+ * <li>{@link #EXACTLY_ONCE}
+ * <li>{@link #AT_LEAST_ONCE}
+ * <li>{@link #NONE}
+ */
+public enum DeliveryGuarantee {
+ EXACTLY_ONCE,
Review comment:
Add JavaDoc and make clear that source+sink need to be EOS.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} TODO: FLINK-23124
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+class KafkaSink<IN> implements Sink<IN, KafkaCommittable, KafkaWriterState, Void> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSink.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Properties kafkaProducerConfig;
+ private final int kafkaProducerPoolSize;
+
+ private KafkaSink(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ int kafkaProducerPoolSize,
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.deliveryGuarantee = deliveryGuarantee;
+ this.kafkaProducerPoolSize = kafkaProducerPoolSize;
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ this.recordSerializer = recordSerializer;
+ }
+
+ @Override
+ public SinkWriter<IN, KafkaCommittable, KafkaWriterState> createWriter(
+ InitContext context, List<KafkaWriterState> states) throws IOException {
+ return new KafkaWriter<>(
+ deliveryGuarantee,
+ kafkaProducerConfig,
+ kafkaProducerPoolSize,
+ context,
+ recordSerializer,
+ new InitContextInitializationContextAdapter(
+ context, metricGroup -> metricGroup.addGroup("user")),
+ states);
+ }
+
+ @Override
+ public Optional<Committer<KafkaCommittable>> createCommitter() throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<GlobalCommitter<KafkaCommittable, Void>> createGlobalCommitter()
+ throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaCommittable>> getCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<Void>> getGlobalCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaWriterState>> getWriterStateSerializer() {
+ return Optional.of(new KafkaWriterStateSerializer(kafkaProducerConfig));
+ }
+
+ /**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+ public static class Builder<IN> {
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private int kafkaProducersPoolSize = 5;
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+
+ public Builder<IN> setSemantic(DeliveryGuarantee deliveryGuarantee) {
Review comment:
`setDeliveryGuarantee`
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} TODO: FLINK-23124
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+class KafkaSink<IN> implements Sink<IN, KafkaCommittable, KafkaWriterState, Void> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSink.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Properties kafkaProducerConfig;
+ private final int kafkaProducerPoolSize;
+
+ private KafkaSink(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ int kafkaProducerPoolSize,
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.deliveryGuarantee = deliveryGuarantee;
+ this.kafkaProducerPoolSize = kafkaProducerPoolSize;
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ this.recordSerializer = recordSerializer;
+ }
+
+ @Override
+ public SinkWriter<IN, KafkaCommittable, KafkaWriterState> createWriter(
+ InitContext context, List<KafkaWriterState> states) throws IOException {
+ return new KafkaWriter<>(
+ deliveryGuarantee,
+ kafkaProducerConfig,
+ kafkaProducerPoolSize,
+ context,
+ recordSerializer,
+ new InitContextInitializationContextAdapter(
+ context, metricGroup -> metricGroup.addGroup("user")),
+ states);
+ }
+
+ @Override
+ public Optional<Committer<KafkaCommittable>> createCommitter() throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<GlobalCommitter<KafkaCommittable, Void>> createGlobalCommitter()
+ throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaCommittable>> getCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<Void>> getGlobalCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaWriterState>> getWriterStateSerializer() {
+ return Optional.of(new KafkaWriterStateSerializer(kafkaProducerConfig));
+ }
+
+ /**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+ public static class Builder<IN> {
Review comment:
Add setBootstrapServers
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} TODO: FLINK-23124
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+class KafkaSink<IN> implements Sink<IN, KafkaCommittable, KafkaWriterState, Void> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSink.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Properties kafkaProducerConfig;
+ private final int kafkaProducerPoolSize;
+
+ private KafkaSink(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ int kafkaProducerPoolSize,
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.deliveryGuarantee = deliveryGuarantee;
+ this.kafkaProducerPoolSize = kafkaProducerPoolSize;
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ this.recordSerializer = recordSerializer;
+ }
+
+ @Override
+ public SinkWriter<IN, KafkaCommittable, KafkaWriterState> createWriter(
+ InitContext context, List<KafkaWriterState> states) throws IOException {
+ return new KafkaWriter<>(
+ deliveryGuarantee,
+ kafkaProducerConfig,
+ kafkaProducerPoolSize,
+ context,
+ recordSerializer,
+ new InitContextInitializationContextAdapter(
+ context, metricGroup -> metricGroup.addGroup("user")),
+ states);
+ }
+
+ @Override
+ public Optional<Committer<KafkaCommittable>> createCommitter() throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<GlobalCommitter<KafkaCommittable, Void>> createGlobalCommitter()
+ throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaCommittable>> getCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<Void>> getGlobalCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaWriterState>> getWriterStateSerializer() {
+ return Optional.of(new KafkaWriterStateSerializer(kafkaProducerConfig));
+ }
+
+ /**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+ public static class Builder<IN> {
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private int kafkaProducersPoolSize = 5;
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+
+ public Builder<IN> setSemantic(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = requireNonNull(deliveryGuarantee, "semantic");
+ return this;
+ }
+
+ public Builder<IN> setKafkaProducerConfig(Properties kafkaProducerConfig) {
+ this.kafkaProducerConfig = requireNonNull(kafkaProducerConfig, "kafkaProducerConfig");
+ // set the producer configuration properties for kafka record key value serializers.
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG)) {
+ kafkaProducerConfig.put(
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
+ ByteArraySerializer.class.getName());
+ } else {
+ LOG.warn(
+ "Overwriting the '{}' is not recommended",
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
+ }
+
+ if (!kafkaProducerConfig.containsKey(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG)) {
+ final long timeout = DEFAULT_KAFKA_TRANSACTION_TIMEOUT.toMillis();
+ checkState(
+ timeout < Integer.MAX_VALUE && timeout > 0,
+ "timeout does not fit into 32 bit integer");
+ kafkaProducerConfig.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, (int) timeout);
+ LOG.warn(
+ "Property [{}] not specified. Setting it to {}",
+ ProducerConfig.TRANSACTION_TIMEOUT_CONFIG,
+ DEFAULT_KAFKA_TRANSACTION_TIMEOUT);
+ }
+ return this;
+ }
+
+ public Builder<IN> setRecordSerializer(
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.recordSerializer = recordSerializer;
Review comment:
checkNotNull
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
Review comment:
Add link to builder.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} TODO: FLINK-23124
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+class KafkaSink<IN> implements Sink<IN, KafkaCommittable, KafkaWriterState, Void> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSink.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Properties kafkaProducerConfig;
+ private final int kafkaProducerPoolSize;
+
+ private KafkaSink(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ int kafkaProducerPoolSize,
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.deliveryGuarantee = deliveryGuarantee;
+ this.kafkaProducerPoolSize = kafkaProducerPoolSize;
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ this.recordSerializer = recordSerializer;
+ }
+
+ @Override
+ public SinkWriter<IN, KafkaCommittable, KafkaWriterState> createWriter(
+ InitContext context, List<KafkaWriterState> states) throws IOException {
+ return new KafkaWriter<>(
+ deliveryGuarantee,
+ kafkaProducerConfig,
+ kafkaProducerPoolSize,
+ context,
+ recordSerializer,
+ new InitContextInitializationContextAdapter(
+ context, metricGroup -> metricGroup.addGroup("user")),
+ states);
+ }
+
+ @Override
+ public Optional<Committer<KafkaCommittable>> createCommitter() throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<GlobalCommitter<KafkaCommittable, Void>> createGlobalCommitter()
+ throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaCommittable>> getCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<Void>> getGlobalCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaWriterState>> getWriterStateSerializer() {
+ return Optional.of(new KafkaWriterStateSerializer(kafkaProducerConfig));
+ }
+
+ /**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+ public static class Builder<IN> {
+
+ private DeliveryGuarantee deliveryGuarantee = DeliveryGuarantee.NONE;
+ private int kafkaProducersPoolSize = 5;
+
+ private Properties kafkaProducerConfig;
+ private KafkaRecordSerializationSchema<IN> recordSerializer;
+
+ public Builder<IN> setSemantic(DeliveryGuarantee deliveryGuarantee) {
+ this.deliveryGuarantee = requireNonNull(deliveryGuarantee, "semantic");
Review comment:
checkNotNull
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} TODO: FLINK-23124
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+class KafkaSink<IN> implements Sink<IN, KafkaCommittable, KafkaWriterState, Void> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSink.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Properties kafkaProducerConfig;
+ private final int kafkaProducerPoolSize;
+
+ private KafkaSink(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ int kafkaProducerPoolSize,
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.deliveryGuarantee = deliveryGuarantee;
+ this.kafkaProducerPoolSize = kafkaProducerPoolSize;
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ this.recordSerializer = recordSerializer;
+ }
+
+ @Override
+ public SinkWriter<IN, KafkaCommittable, KafkaWriterState> createWriter(
+ InitContext context, List<KafkaWriterState> states) throws IOException {
+ return new KafkaWriter<>(
+ deliveryGuarantee,
+ kafkaProducerConfig,
+ kafkaProducerPoolSize,
+ context,
+ recordSerializer,
+ new InitContextInitializationContextAdapter(
+ context, metricGroup -> metricGroup.addGroup("user")),
+ states);
+ }
+
+ @Override
+ public Optional<Committer<KafkaCommittable>> createCommitter() throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<GlobalCommitter<KafkaCommittable, Void>> createGlobalCommitter()
+ throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaCommittable>> getCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<Void>> getGlobalCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaWriterState>> getWriterStateSerializer() {
+ return Optional.of(new KafkaWriterStateSerializer(kafkaProducerConfig));
+ }
+
+ /**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+ public static class Builder<IN> {
Review comment:
Pull out Builder as KafkaSinkBuilder.
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} TODO: FLINK-23124
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+class KafkaSink<IN> implements Sink<IN, KafkaCommittable, KafkaWriterState, Void> {
Review comment:
+public
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkITCase.java
##########
@@ -0,0 +1,659 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.StateBackendOptions;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.testutils.junit.SharedReference;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.ImmutableMap;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.CreateTopicsResult;
+import org.apache.kafka.clients.admin.DeleteTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.LongStream;
+
+import static org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorageAccess.CHECKPOINT_DIR_PREFIX;
+import static org.apache.flink.runtime.state.filesystem.AbstractFsCheckpointStorageAccess.METADATA_FILE_NAME;
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+/** Tests for using KafkaSink writing to a Kafka cluster. */
+public class KafkaSinkITCase extends TestLogger {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkITCase.class);
+ private static final Slf4jLogConsumer LOG_CONSUMER = new Slf4jLogConsumer(LOG);
+ private static final String INTER_CONTAINER_KAFKA_ALIAS = "kafka";
+ private static final Network NETWORK = Network.newNetwork();
+ private static final int ZK_TIMEOUT_MILLIS = 30000;
+ private static final short TOPIC_REPLICATION_FACTOR = 1;
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+
+ private String topic;
+ private SharedReference<AtomicLong> emittedRecordsCount;
+ private SharedReference<AtomicLong> emittedRecordsWithCheckpoint;
+ private SharedReference<AtomicBoolean> failed;
+ private SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ @ClassRule
+ public static final KafkaContainer KAFKA_CONTAINER =
+ new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.5.2"))
+ .withEmbeddedZookeeper()
+ .withEnv(
+ ImmutableMap.of(
+ "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR",
+ "1",
+ "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
+ String.valueOf(Duration.ofHours(2).toMillis()),
+ "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR",
+ "1",
+ "KAFKA_MIN_INSYNC_REPLICAS",
+ "1"))
+ .withNetwork(NETWORK)
+ .withLogConsumer(LOG_CONSUMER)
+ .withNetworkAliases(INTER_CONTAINER_KAFKA_ALIAS);
+
+ @Rule public final SharedObjects sharedObjects = SharedObjects.create();
+
+ @Rule public final TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void setUp() throws ExecutionException, InterruptedException, TimeoutException {
+ emittedRecordsCount = sharedObjects.add(new AtomicLong());
+ emittedRecordsWithCheckpoint = sharedObjects.add(new AtomicLong());
+ failed = sharedObjects.add(new AtomicBoolean(false));
+ lastCheckpointedRecord = sharedObjects.add(new AtomicLong(0));
+ topic = UUID.randomUUID().toString();
+ createTestTopic(topic, 1, TOPIC_REPLICATION_FACTOR);
+ }
+
+ @After
+ public void tearDown() throws ExecutionException, InterruptedException, TimeoutException {
+ deleteTestTopic(topic);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithAtLeastOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.AT_LEAST_ONCE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithNoneGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.NONE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithExactlyOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.EXACTLY_ONCE, emittedRecordsWithCheckpoint);
+ }
+
+ @Test
+ public void testRecoveryWithAtLeastOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.AT_LEAST_ONCE,
+ (records) ->
+ assertThat(records, hasItems(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L)));
+ }
+
+ @Test
+ public void testRecoveryWithExactlyOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.EXACTLY_ONCE,
+ (records) ->
+ assertEquals(
+ records,
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList())));
+ }
+
+ @Test
+ public void testAbortTransactionsOfPendingCheckpointsAfterFailure() throws Exception {
+ // Run a first job failing during the async phase of a checkpoint to leave some
+ // lingering transactions
+ final Configuration config = new Configuration();
+ config.setString(StateBackendOptions.STATE_BACKEND, "filesystem");
+ final File checkpointDir = temp.newFolder();
+ config.setString(
+ CheckpointingOptions.CHECKPOINTS_DIRECTORY, checkpointDir.toURI().toString());
+ config.set(
+ ExecutionCheckpointingOptions.EXTERNALIZED_CHECKPOINT,
+ CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
+ config.set(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 2);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(1), config, "firstPrefix");
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ final File completedCheckpoint =
+ Files.find(checkpointDir.toPath(), 2, this::isCompletedCheckpoint)
+ .max(Comparator.comparing(Path::toString))
+ .map(Path::toFile)
+ .orElseThrow(() -> new IllegalStateException("Cannot generate checkpoint"));
Review comment:
Extract `org.apache.flink.test.util.TestUtils`
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSink.java
##########
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.InitContextInitializationContextAdapter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.api.java.ClosureCleaner;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees
+ * described by {@link DeliveryGuarantee}.
+ * <li>{@link DeliveryGuarantee#NONE} is it not guaranteed that no messages are lost and in case of
+ * a failure messages may be duplicated.
+ * <li>{@link DeliveryGuarantee#AT_LEAST_ONCE} the sink will wait for all outstanding records in the
+ * Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
+ * <li>{@link DeliveryGuarantee#EXACTLY_ONCE} TODO: FLINK-23124
+ *
+ * <p>When creating the sink it is required to specify {@code deliveryGuarantee}, {@link
+ * Properties} to configure the Kafka Producer and one has to implement the {@link
+ * KafkaRecordSerializationSchema}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+class KafkaSink<IN> implements Sink<IN, KafkaCommittable, KafkaWriterState, Void> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSink.class);
+ private static final Duration DEFAULT_KAFKA_TRANSACTION_TIMEOUT = Duration.ofHours(1);
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Properties kafkaProducerConfig;
+ private final int kafkaProducerPoolSize;
+
+ private KafkaSink(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ int kafkaProducerPoolSize,
+ KafkaRecordSerializationSchema<IN> recordSerializer) {
+ this.deliveryGuarantee = deliveryGuarantee;
+ this.kafkaProducerPoolSize = kafkaProducerPoolSize;
+ this.kafkaProducerConfig = kafkaProducerConfig;
+ this.recordSerializer = recordSerializer;
+ }
+
+ @Override
+ public SinkWriter<IN, KafkaCommittable, KafkaWriterState> createWriter(
+ InitContext context, List<KafkaWriterState> states) throws IOException {
+ return new KafkaWriter<>(
+ deliveryGuarantee,
+ kafkaProducerConfig,
+ kafkaProducerPoolSize,
+ context,
+ recordSerializer,
+ new InitContextInitializationContextAdapter(
+ context, metricGroup -> metricGroup.addGroup("user")),
+ states);
+ }
+
+ @Override
+ public Optional<Committer<KafkaCommittable>> createCommitter() throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<GlobalCommitter<KafkaCommittable, Void>> createGlobalCommitter()
+ throws IOException {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaCommittable>> getCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<Void>> getGlobalCommittableSerializer() {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<SimpleVersionedSerializer<KafkaWriterState>> getWriterStateSerializer() {
+ return Optional.of(new KafkaWriterStateSerializer(kafkaProducerConfig));
+ }
+
+ /**
+ * Builder to construct {@link KafkaSink}.
+ *
+ * @param <IN> type of the records written to Kafka
+ */
+ public static class Builder<IN> {
Review comment:
Add KafkaSink#builder()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
}, {
"hash" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"triggerType" : "PUSH"
}, {
"hash" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681",
"triggerID" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* e8c6ec06f2263f5704ccf597494c43a1bd0e5893 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664)
* 5f636910791eb5ddaa6b9b210c3a4b3230fc60bd UNKNOWN
* 055d0fd83c72f141a1bd2d8a4bc0e30bf637687f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684076946
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriter.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.Metric;
+import org.apache.kafka.common.MetricName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is responsible to write records in a Kafka topic and to handle the different delivery
+ * {@link DeliveryGuarantee}s.
+ *
+ * @param <IN> The type of the input elements.
+ */
+class KafkaWriter<IN> implements SinkWriter<IN, KafkaCommittable, KafkaWriterState> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaWriter.class);
+ private static final String KEY_DISABLE_METRICS = "flink.disable-metrics";
+ private static final String KAFKA_PRODUCER_METRIC_NAME = "KafkaProducer";
+
+ private final DeliveryGuarantee deliveryGuarantee;
+ private final Properties kafkaProducerConfig;
+ private final String transactionalIdPrefix;
+ private final KafkaRecordSerializationSchema<IN> recordSerializer;
+ private final Callback deliveryCallback;
+ private final AtomicLong pendingRecords = new AtomicLong();
+ private final KafkaRecordSerializationSchema.KafkaSinkContext kafkaSinkContext;
+ private final List<FlinkKafkaInternalProducer<byte[], byte[]>> producers = new ArrayList<>();
+ private final Map<String, KafkaMetricMutableWrapper> previouslyCreatedMetrics = new HashMap<>();
+ private final MetricGroup metricGroup;
+
+ private transient FlinkKafkaInternalProducer<byte[], byte[]> currentProducer;
+ private transient KafkaWriterState kafkaWriterState;
+ @Nullable private transient volatile Exception producerAsyncException;
+
+ /**
+ * Constructor creating a kafka writer.
+ *
+ * <p>It will throw a {@link RuntimeException} if {@link
+ * KafkaRecordSerializationSchema#open(SerializationSchema.InitializationContext)} fails.
+ *
+ * @param deliveryGuarantee the Sink's delivery guarantee
+ * @param kafkaProducerConfig the properties to configure the {@link FlinkKafkaInternalProducer}
+ * @param transactionalIdPrefix used to create the transactionalIds
+ * @param sinkInitContext context to provide information about the runtime environment
+ * @param recordSerializer serialize to transform the incoming records to {@link ProducerRecord}
+ * @param schemaContext context used to initialize the {@link KafkaRecordSerializationSchema}
+ * @param recoveredStates state from an previous execution which was covered
+ */
+ KafkaWriter(
+ DeliveryGuarantee deliveryGuarantee,
+ Properties kafkaProducerConfig,
+ String transactionalIdPrefix,
+ Sink.InitContext sinkInitContext,
+ KafkaRecordSerializationSchema<IN> recordSerializer,
+ SerializationSchema.InitializationContext schemaContext,
+ List<KafkaWriterState> recoveredStates) {
+ this.deliveryGuarantee = checkNotNull(deliveryGuarantee, "deliveryGuarantee");
+ this.kafkaProducerConfig = checkNotNull(kafkaProducerConfig, "kafkaProducerConfig");
+ this.transactionalIdPrefix = checkNotNull(transactionalIdPrefix, "transactionalIdPrefix");
+ this.recordSerializer = checkNotNull(recordSerializer, "recordSerializer");
+ try {
+ recordSerializer.open(schemaContext);
+ } catch (Exception e) {
+ throw new RuntimeException("Cannot initialize schema.", e);
+ }
+ this.deliveryCallback =
+ (metadata, exception) -> {
+ if (exception != null && producerAsyncException == null) {
+ producerAsyncException = exception;
+ }
+ acknowledgeMessage();
+ };
+ this.metricGroup = sinkInitContext.metricGroup();
+ this.kafkaSinkContext =
+ new DefaultKafkaSinkContext(
+ checkNotNull(sinkInitContext, "sinkInitContext"), kafkaProducerConfig);
+ this.kafkaWriterState =
+ recoverAndInitializeState(checkNotNull(recoveredStates, "recoveredStates"));
+ this.currentProducer = beginTransaction();
+ producers.add(currentProducer);
+ }
+
+ @Override
+ public void write(IN element, Context context) throws IOException {
+ checkErroneous();
+ final ProducerRecord<byte[], byte[]> record =
+ recordSerializer.serialize(element, kafkaSinkContext, context.timestamp());
+ pendingRecords.incrementAndGet();
+ currentProducer.send(record, deliveryCallback);
+ }
+
+ @Override
+ public List<KafkaCommittable> prepareCommit(boolean flush) throws IOException {
+ flushRecords(flush);
+ if (!flush) {
+ currentProducer = beginTransaction();
+ }
+ final List<KafkaCommittable> committables = commit();
+ producers.add(currentProducer);
+ return committables;
+ }
+
+ @Override
+ public List<KafkaWriterState> snapshotState() throws IOException {
+ return ImmutableList.of(kafkaWriterState);
+ }
+
+ @Override
+ public void close() throws Exception {
+ currentProducer.close(Duration.ZERO);
Review comment:
Every time a new producer is created we immediately close the old thus we do not need to do it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a4a37916fdd2d49a7682b2d90da22e5ed398eae5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684095673
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkITCase.java
##########
@@ -0,0 +1,634 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.StateBackendOptions;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.test.util.TestUtils;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.testutils.junit.SharedReference;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.CreateTopicsResult;
+import org.apache.kafka.clients.admin.DeleteTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.LongStream;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+/** Tests for using KafkaSink writing to a Kafka cluster. */
+public class KafkaSinkITCase extends TestLogger {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkITCase.class);
+ private static final Slf4jLogConsumer LOG_CONSUMER = new Slf4jLogConsumer(LOG);
+ private static final String INTER_CONTAINER_KAFKA_ALIAS = "kafka";
+ private static final Network NETWORK = Network.newNetwork();
+ private static final int ZK_TIMEOUT_MILLIS = 30000;
+ private static final short TOPIC_REPLICATION_FACTOR = 1;
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+
+ private String topic;
+ private SharedReference<AtomicLong> emittedRecordsCount;
+ private SharedReference<AtomicLong> emittedRecordsWithCheckpoint;
+ private SharedReference<AtomicBoolean> failed;
+ private SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ @ClassRule
+ public static final KafkaContainer KAFKA_CONTAINER =
+ new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.5.2"))
+ .withEmbeddedZookeeper()
+ .withEnv(
+ ImmutableMap.of(
+ "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR",
+ "1",
+ "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
+ String.valueOf(Duration.ofHours(2).toMillis()),
+ "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR",
+ "1",
+ "KAFKA_MIN_INSYNC_REPLICAS",
+ "1"))
+ .withNetwork(NETWORK)
+ .withLogConsumer(LOG_CONSUMER)
+ .withNetworkAliases(INTER_CONTAINER_KAFKA_ALIAS);
+
+ @Rule public final SharedObjects sharedObjects = SharedObjects.create();
+
+ @Rule public final TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void setUp() throws ExecutionException, InterruptedException, TimeoutException {
+ emittedRecordsCount = sharedObjects.add(new AtomicLong());
+ emittedRecordsWithCheckpoint = sharedObjects.add(new AtomicLong());
+ failed = sharedObjects.add(new AtomicBoolean(false));
+ lastCheckpointedRecord = sharedObjects.add(new AtomicLong(0));
+ topic = UUID.randomUUID().toString();
+ createTestTopic(topic, 1, TOPIC_REPLICATION_FACTOR);
+ }
+
+ @After
+ public void tearDown() throws ExecutionException, InterruptedException, TimeoutException {
+ deleteTestTopic(topic);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithAtLeastOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.AT_LEAST_ONCE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithNoneGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.NONE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithExactlyOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.EXACTLY_ONCE, emittedRecordsWithCheckpoint);
+ }
+
+ @Test
+ public void testRecoveryWithAtLeastOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.AT_LEAST_ONCE,
+ (records) ->
+ assertThat(records, hasItems(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L)));
+ }
+
+ @Test
+ public void testRecoveryWithExactlyOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.EXACTLY_ONCE,
+ (records) ->
+ assertEquals(
+ records,
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList())));
+ }
+
+ @Test
+ public void testAbortTransactionsOfPendingCheckpointsAfterFailure() throws Exception {
+ // Run a first job failing during the async phase of a checkpoint to leave some
+ // lingering transactions
+ final Configuration config = new Configuration();
+ config.setString(StateBackendOptions.STATE_BACKEND, "filesystem");
+ final File checkpointDir = temp.newFolder();
+ config.setString(
+ CheckpointingOptions.CHECKPOINTS_DIRECTORY, checkpointDir.toURI().toString());
+ config.set(
+ ExecutionCheckpointingOptions.EXTERNALIZED_CHECKPOINT,
+ CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
+ config.set(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 2);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(1), config, "firstPrefix");
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ final File completedCheckpoint = TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
+
+ config.set(SavepointConfigOptions.SAVEPOINT_PATH, completedCheckpoint.toURI().toString());
+
+ // Run a second job which aborts all lingering transactions and new consumer should
+ // immediately see the newly written records
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, "newPrefix");
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ @Test
+ public void testAbortTransactionsAfterScaleInBeforeFirstCheckpoint() throws Exception {
+ // Run a first job opening 5 transactions one per subtask and fail in async checkpoint phase
+ final Configuration config = new Configuration();
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 5);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(0), config, null);
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ assertTrue(deserializeValues(drainAllRecordsFromTopic(topic)).isEmpty());
+
+ // Second job aborts all transactions from previous runs with higher parallelism
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 1);
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, null);
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ private void executeWithMapper(
+ MapFunction<Long, Long> mapper,
+ Configuration config,
+ @Nullable String transactionalIdPrefix)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment(config);
+ env.enableCheckpointing(100L);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ final DataStreamSource<Long> source = env.fromSequence(1, 10);
+ final DataStream<Long> stream = source.map(mapper);
+ final KafkaSinkBuilder<Long> builder =
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic));
+ if (transactionalIdPrefix == null) {
+ transactionalIdPrefix = "kafka-sink";
+ }
+ builder.setTransactionalIdPrefix(transactionalIdPrefix);
+ stream.sinkTo(builder.build());
+ env.execute();
+ checkProducerLeak();
+ }
+
+ private void testRecoveryWithAssertion(
+ DeliveryGuarantee guarantee, java.util.function.Consumer<List<Long>> recordsAssertion)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment();
+ env.enableCheckpointing(300L);
+ DataStreamSource<Long> source = env.fromSequence(1, 10);
+ DataStream<Long> stream =
+ source.map(new FailingCheckpointMapper(failed, lastCheckpointedRecord));
+
+ stream.sinkTo(
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(guarantee)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic))
+ .setTransactionalIdPrefix("kafka-sink")
+ .build());
+ env.execute();
+
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ recordsAssertion.accept(deserializeValues(collectedRecords));
+ checkProducerLeak();
+ }
+
+ private void writeRecordsToKafka(
+ DeliveryGuarantee deliveryGuarantee, SharedReference<AtomicLong> expectedRecords)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment();
+ env.enableCheckpointing(1000L);
+ final DataStream<Long> source =
+ env.addSource(
+ new InfiniteIntegerSource(
+ emittedRecordsCount, emittedRecordsWithCheckpoint));
+ source.sinkTo(
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setDeliverGuarantee(deliveryGuarantee)
+ .setRecordSerializer(new RecordSerializer(topic))
+ .setTransactionalIdPrefix("kafka-sink")
+ .build());
+ env.execute();
+
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ final long recordsCount = expectedRecords.get().get();
+ assertEquals(collectedRecords.size(), recordsCount);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(0, recordsCount).boxed().collect(Collectors.toList()));
+ checkProducerLeak();
+ }
+
+ private static List<Long> deserializeValues(List<ConsumerRecord<byte[], byte[]>> records) {
+ return records.stream()
+ .map(
+ record -> {
+ final ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ final byte[] value = record.value();
+ buffer.put(value, 0, value.length);
+ buffer.flip();
+ return buffer.getLong();
+ })
+ .collect(Collectors.toList());
+ }
+
+ private static Properties getKafkaClientConfiguration() {
+ final Properties standardProps = new Properties();
+ standardProps.put("bootstrap.servers", KAFKA_CONTAINER.getBootstrapServers());
+ standardProps.put("group.id", UUID.randomUUID().toString());
+ standardProps.put("enable.auto.commit", false);
+ standardProps.put("auto.offset.reset", "earliest");
+ standardProps.put("max.partition.fetch.bytes", 256);
+ standardProps.put("zookeeper.session.timeout.ms", ZK_TIMEOUT_MILLIS);
+ standardProps.put("zookeeper.connection.timeout.ms", ZK_TIMEOUT_MILLIS);
+ return standardProps;
+ }
+
+ private Consumer<byte[], byte[]> createTestConsumer(String topic) {
+ final Properties consumerConfig = new Properties();
+ consumerConfig.putAll(getKafkaClientConfiguration());
+ consumerConfig.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ consumerConfig.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ consumerConfig.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ final KafkaConsumer<byte[], byte[]> kafkaConsumer = new KafkaConsumer<>(consumerConfig);
+ kafkaConsumer.subscribe(Collections.singletonList(topic));
+ return kafkaConsumer;
+ }
+
+ private void createTestTopic(String topic, int numPartitions, short replicationFactor)
+ throws ExecutionException, InterruptedException, TimeoutException {
+ Map<String, Object> properties = new HashMap<>();
+ properties.put(
+ CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
+ KAFKA_CONTAINER.getBootstrapServers());
+ try (AdminClient admin = AdminClient.create(properties)) {
+ final CreateTopicsResult result =
+ admin.createTopics(
+ Collections.singletonList(
+ new NewTopic(topic, numPartitions, replicationFactor)));
+ result.all().get(1, TimeUnit.MINUTES);
+ }
+ }
+
+ private void deleteTestTopic(String topic)
+ throws ExecutionException, InterruptedException, TimeoutException {
+ Map<String, Object> properties = new HashMap<>();
+ properties.put(
+ CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
+ KAFKA_CONTAINER.getBootstrapServers());
+ try (AdminClient admin = AdminClient.create(properties)) {
+ final DeleteTopicsResult result = admin.deleteTopics(Collections.singletonList(topic));
+ result.all().get(1, TimeUnit.MINUTES);
+ }
+ }
+
+ private List<ConsumerRecord<byte[], byte[]>> drainAllRecordsFromTopic(String topic) {
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords = new ArrayList<>();
+ try (Consumer<byte[], byte[]> consumer = createTestConsumer(topic)) {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ // Drain the kafka topic till all records are consumed
+ while (!records.isEmpty()) {
+ records.records(topic).forEach(collectedRecords::add);
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ }
+ return collectedRecords;
+ }
+
+ private static class RecordSerializer implements KafkaRecordSerializationSchema<Long> {
+
+ private final String topic;
+
+ public RecordSerializer(String topic) {
+ this.topic = topic;
+ }
+
+ @Override
+ public ProducerRecord<byte[], byte[]> serialize(
+ Long element, KafkaSinkContext context, Long timestamp) {
+ final ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ buffer.putLong(element);
+ return new ProducerRecord<>(topic, 0, null, null, buffer.array());
+ }
+ }
+
+ private static class FailAsyncCheckpointMapper
+ implements MapFunction<Long, Long>, CheckpointedFunction {
+ private static final ListStateDescriptor<Integer> stateDescriptor =
+ new ListStateDescriptor<>("test-state", new SlowSerializer());
+ private int failAfterCheckpoint;
+
+ private ListState<Integer> state;
+
+ public FailAsyncCheckpointMapper(int failAfterCheckpoint) {
+ this.failAfterCheckpoint = failAfterCheckpoint;
+ }
+
+ @Override
+ public Long map(Long value) throws Exception {
+ Thread.sleep(100);
+ return value;
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ state.clear();
+ if (failAfterCheckpoint <= 0) {
+ // Trigger a failure in the serializer
+ state.add(-1);
+ } else {
+ state.add(1);
+ }
+ failAfterCheckpoint--;
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ state = context.getOperatorStateStore().getListState(stateDescriptor);
+ }
+ }
+
+ private static class SlowSerializer extends TypeSerializerSingleton<Integer> {
+
+ @Override
+ public boolean isImmutableType() {
+ return false;
+ }
+
+ @Override
+ public Integer createInstance() {
+ return 1;
+ }
+
+ @Override
+ public Integer copy(Integer from) {
+ return from;
+ }
+
+ @Override
+ public Integer copy(Integer from, Integer reuse) {
+ return from;
+ }
+
+ @Override
+ public int getLength() {
+ return 0;
+ }
+
+ @Override
+ public void serialize(Integer record, DataOutputView target) throws IOException {
+ if (record != -1) {
+ return;
+ }
+ throw new RuntimeException("Expected failure during async checkpoint phase");
+ }
+
+ @Override
+ public Integer deserialize(DataInputView source) throws IOException {
+ return 1;
+ }
+
+ @Override
+ public Integer deserialize(Integer reuse, DataInputView source) throws IOException {
+ return 1;
+ }
+
+ @Override
+ public void copy(DataInputView source, DataOutputView target) throws IOException {}
+
+ @Override
+ public TypeSerializerSnapshot<Integer> snapshotConfiguration() {
+ return new SlowSerializerSnapshot();
+ }
+ }
+
+ /** Snapshot used in {@link FailAsyncCheckpointMapper}. */
+ public static class SlowSerializerSnapshot extends SimpleTypeSerializerSnapshot<Integer> {
+ public SlowSerializerSnapshot() {
+ super(SlowSerializer::new);
+ }
+ }
+
+ /** Fails after a checkpoint is taken and the next record was emitted. */
+ private static class FailingCheckpointMapper
+ implements MapFunction<Long, Long>, CheckpointListener, CheckpointedFunction {
+
+ private final SharedReference<AtomicBoolean> failed;
+ private final SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ private volatile long lastSeenRecord;
+ private volatile long checkpointedRecord;
+ private volatile long lastCheckpointId = 0;
+ private final AtomicInteger emittedBetweenCheckpoint = new AtomicInteger(0);
+
+ FailingCheckpointMapper(
+ SharedReference<AtomicBoolean> failed,
+ SharedReference<AtomicLong> lastCheckpointedRecord) {
+ this.failed = failed;
+ this.lastCheckpointedRecord = lastCheckpointedRecord;
+ }
+
+ @Override
+ public Long map(Long value) throws Exception {
+ lastSeenRecord = value;
+ if (lastCheckpointId >= 1
+ && emittedBetweenCheckpoint.get() > 0
+ && !failed.get().get()) {
+ failed.get().set(true);
+ throw new RuntimeException("Planned exception.");
+ }
+ Thread.sleep(50);
+ emittedBetweenCheckpoint.incrementAndGet();
+ return value;
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ lastCheckpointId = checkpointId;
+ emittedBetweenCheckpoint.set(0);
+ lastCheckpointedRecord.get().set(checkpointedRecord);
+ }
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) throws Exception {
+ checkpointedRecord = lastSeenRecord;
+ }
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {}
+ }
+
+ private void checkProducerLeak() {
+ for (Thread t : Thread.getAllStackTraces().keySet()) {
+ if (t.getName().contains("kafka-producer-network-thread")) {
+ fail("Detected producer leak. Thread name: " + t.getName());
+ }
+ }
+ }
+
+ /**
+ * Exposes information about how man records have been emitted overall and at the beginning of a
+ * checkpoint.
+ */
+ private static final class InfiniteIntegerSource
Review comment:
I do not see how I can easily replace it because I am relying on the fact that the Source finishes after the first `checkpointCompleted` event.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
}, {
"hash" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"triggerType" : "PUSH"
}, {
"hash" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681",
"triggerID" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"triggerType" : "PUSH"
}, {
"hash" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"triggerType" : "PUSH"
}, {
"hash" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21692",
"triggerID" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"triggerType" : "PUSH"
}, {
"hash" : "483a1a6e982e43b538f13152d41b71144143e444",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21701",
"triggerID" : "483a1a6e982e43b538f13152d41b71144143e444",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 5f636910791eb5ddaa6b9b210c3a4b3230fc60bd UNKNOWN
* 96995fe1f26d7a08e91308409d2c66ec32ec8819 UNKNOWN
* 2d1d736dc15e4b96b13a917ecfdd9716d8d48be4 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21692)
* 483a1a6e982e43b538f13152d41b71144143e444 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21701)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25fb6f9a84ab137be93a04bfad41c4ab021a3b4b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464)
* 1e41696e78086dda1970f6e50e3125d0271c056b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494)
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 77d980b81bcd178befe0a1f3762234a2e5b79222 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684150895
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/TransactionalIdFactory.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.base.Splitter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+class TransactionalIdFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TransactionalIdFactory.class);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
Review comment:
Hmm, it means we need to instantiate the factory basically for every transaction. What would be the benefit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e41696e78086dda1970f6e50e3125d0271c056b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494)
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 77d980b81bcd178befe0a1f3762234a2e5b79222 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520)
* 3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531)
* 5916417a1266f4af34d387dd2e61a10673a1a9fe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 77d980b81bcd178befe0a1f3762234a2e5b79222 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520)
* 3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531)
* 5916417a1266f4af34d387dd2e61a10673a1a9fe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r683293143
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/FlinkKafkaInternalProducer.java
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.util.Preconditions;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.TransactionalRequestResult;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Properties;
+
+/**
+ * A {@link KafkaProducer} that exposes private fields to allow resume producing from a given state.
+ */
+class FlinkKafkaInternalProducer<K, V> extends KafkaProducer<K, V> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkKafkaInternalProducer.class);
+ private static final String TRANSACTION_MANAGER_STATE_ENUM =
+ "org.apache.kafka.clients.producer.internals.TransactionManager$State";
+
+ private final Properties kafkaProducerConfig;
+ @Nullable private final String transactionalId;
+
+ public FlinkKafkaInternalProducer(Properties properties) {
+ super(properties);
+ this.kafkaProducerConfig = properties;
+ this.transactionalId = properties.getProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG);
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ if (transactionalId != null) {
+ flushNewPartitions();
+ }
+ }
+
+ public Properties getKafkaProducerConfig() {
+ return kafkaProducerConfig;
+ }
+
+ public short getEpoch() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (short) getField(producerIdAndEpoch, "epoch");
+ }
+
+ public long getProducerId() {
+ Object transactionManager = getField("transactionManager");
+ Object producerIdAndEpoch = getField(transactionManager, "producerIdAndEpoch");
+ return (long) getField(producerIdAndEpoch, "producerId");
+ }
+
+ /**
+ * Besides committing {@link org.apache.kafka.clients.producer.KafkaProducer#commitTransaction}
+ * is also adding new partitions to the transaction. flushNewPartitions method is moving this
+ * logic to pre-commit/flush, to make resumeTransaction simpler. Otherwise resumeTransaction
+ * would require to restore state of the not yet added/"in-flight" partitions.
+ */
+ private void flushNewPartitions() {
+ LOG.info("Flushing new partitions");
+ TransactionalRequestResult result = enqueueNewPartitions();
+ Object sender = getField("sender");
+ invoke(sender, "wakeup");
+ result.await();
+ }
+
+ /**
+ * Enqueues new transactions at the transaction manager and returns a {@link
+ * TransactionalRequestResult} that allows waiting on them.
+ *
+ * <p>If there are no new transactions we return a {@link TransactionalRequestResult} that is
+ * already done.
+ */
+ private TransactionalRequestResult enqueueNewPartitions() {
+ Object transactionManager = getField("transactionManager");
+ synchronized (transactionManager) {
+ Object newPartitionsInTransaction =
+ getField(transactionManager, "newPartitionsInTransaction");
+ Object newPartitionsInTransactionIsEmpty =
+ invoke(newPartitionsInTransaction, "isEmpty");
+ TransactionalRequestResult result;
+ if (newPartitionsInTransactionIsEmpty instanceof Boolean
+ && !((Boolean) newPartitionsInTransactionIsEmpty)) {
+ Object txnRequestHandler =
+ invoke(transactionManager, "addPartitionsToTransactionHandler");
+ invoke(
+ transactionManager,
+ "enqueueRequest",
+ new Class[] {txnRequestHandler.getClass().getSuperclass()},
+ new Object[] {txnRequestHandler});
+ result =
+ (TransactionalRequestResult)
+ getField(
+ txnRequestHandler,
+ txnRequestHandler.getClass().getSuperclass(),
+ "result");
+ } else {
+ // we don't have an operation but this operation string is also used in
+ // addPartitionsToTransactionHandler.
+ result = new TransactionalRequestResult("AddPartitionsToTxn");
+ result.done();
+ }
+ return result;
+ }
+ }
+
+ private static Object invoke(Object object, String methodName, Object... args) {
+ Class<?>[] argTypes = new Class[args.length];
+ for (int i = 0; i < args.length; i++) {
+ argTypes[i] = args[i].getClass();
+ }
+ return invoke(object, methodName, argTypes, args);
+ }
+
+ private static Object invoke(
+ Object object, String methodName, Class<?>[] argTypes, Object[] args) {
+ try {
+ Method method = object.getClass().getDeclaredMethod(methodName, argTypes);
+ method.setAccessible(true);
+ return method.invoke(object, args);
+ } catch (NoSuchMethodException | InvocationTargetException | IllegalAccessException e) {
+ throw new RuntimeException("Incompatible KafkaProducer version", e);
+ }
+ }
Review comment:
Can you give an example of what exactly you want to see here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684138076
##########
File path: flink-core/src/main/java/org/apache/flink/api/connector/sink/InitContextInitializationContextAdapter.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.connector.sink;
+
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import java.util.function.Function;
+
+/**
+ * Adapter between {@link Sink.InitContext} and {@link SerializationSchema.InitializationContext}.
+ */
+public class InitContextInitializationContextAdapter
Review comment:
I wonder if this should be in flink-connector-base instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6287ea41e01d215a3974dca0ae037537a7f3d4bb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397)
* a4a37916fdd2d49a7682b2d90da22e5ed398eae5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22e7bb8757f57435a056f4dd720659c5a3207339 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e41696e78086dda1970f6e50e3125d0271c056b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494)
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 77d980b81bcd178befe0a1f3762234a2e5b79222 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r683431798
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaTransactionLog.java
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableSet;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.Properties;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Consumer;
+
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteAbort;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.CompleteCommit;
+import static org.apache.flink.streaming.connectors.kafka.sink.KafkaTransactionLog.TransactionState.Dead;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.kafka.common.internals.Topic.TRANSACTION_STATE_TOPIC_NAME;
+
+/**
+ * This class is responsible to provide the format of the used transationalIds and in case of an
+ * application restart query the open transactions and decide which must be aborted.
+ */
+class KafkaTransactionLog implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaTransactionLog.class);
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+ private static final Set<TransactionState> TERMINAL_TRANSACTION_STATES =
+ ImmutableSet.of(CompleteCommit, CompleteAbort, Dead);
+ private static final String TRANSACTIONAL_ID_DELIMITER = "-";
+ private static final int SUPPORTED_KAFKA_SCHEMA_VERSION = 0;
+
+ private final KafkaConsumer<byte[], byte[]> consumer;
+ private final KafkaWriterState main;
+ private final int numberOfParallelSubtasks;
+ private final List<KafkaWriterState> combined;
+ private final OptionalInt minOthers;
+
+ /**
+ * Constructor creating a KafkaTransactionLog.
+ *
+ * @param kafkaConfig used to configure the {@link KafkaConsumer} to query the topic containing
+ * the transaction information
+ * @param main the {@link KafkaWriterState} which was previously snapshotted by this subtask
+ * @param others the {@link KafkaWriterState}s which are from different subtasks i.e. in case of
+ * a scale-in
+ * @param numberOfParallelSubtasks current number of parallel sink tasks
+ */
+ KafkaTransactionLog(
+ Properties kafkaConfig,
+ KafkaWriterState main,
+ List<KafkaWriterState> others,
+ int numberOfParallelSubtasks) {
+ this.main = checkNotNull(main, "mainState");
+ checkNotNull(others, "othersState");
+ this.minOthers =
+ others.stream()
+ .map(KafkaWriterState::getSubtaskId)
+ .mapToInt(Integer::intValue)
+ .min();
+ this.combined =
+ new ImmutableList.Builder<KafkaWriterState>().add(main).addAll(others).build();
+ this.numberOfParallelSubtasks = numberOfParallelSubtasks;
+ final Properties copy = new Properties();
+ copy.putAll(checkNotNull(kafkaConfig, "kafkaConfig"));
+ copy.put("enable.auto.commit", false);
+ copy.put("key.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString());
+ copy.put("auto.offset.reset", "earliest");
+ copy.put("value.deserializer", ByteArrayDeserializer.class.getName());
+ copy.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
+ this.consumer = new KafkaConsumer<>(copy);
+ this.consumer.subscribe(ImmutableList.of(TRANSACTION_STATE_TOPIC_NAME));
+ }
+
+ /**
+ * This method queries Kafka's internal transaction topic and filters the transactions for the
+ * following rules.
+ * <li>transaction is in no terminal state {@link
+ * KafkaTransactionLog#TERMINAL_TRANSACTION_STATES}
+ * <li>transactionalIdPrefix equals the one from {@link #main}
+ * <li>Either [transaction's subtaskId % {@link #numberOfParallelSubtasks} = {@link #main}'s
+ * subtaskId] or [transaction's subtaskId == {@link #combined}'s subtaskId && transaction's
+ * checkpointOffset >= {@link #combined}'s checkpointOffset] or [transaction's subtaskId >
+ * {@link #minOthers} && transaction's checkpointOffset == 0]
+ *
+ * @return all transactionIds which must be aborted before starting new transactions.
+ */
+ public Set<String> getTransactionsToAbort() {
+ ConsumerRecords<byte[], byte[]> records = consumer.poll(CONSUMER_POLL_DURATION);
+ final Set<String> openTransactions = new HashSet<>();
+ while (!records.isEmpty()) {
+ records.records(TRANSACTION_STATE_TOPIC_NAME)
+ .forEach(maybeAddTransaction(openTransactions));
+ records = consumer.poll(CONSUMER_POLL_DURATION);
+ }
+ return openTransactions;
+ }
+
+ /**
+ * Constructs a transactionalId with the following format {@code
+ * transactionalIdPrefix-subtaskId-offset}.
+ *
+ * @param transactionalIdPrefix prefix for the id
+ * @param subtaskId describing the subtask which is opening the transaction
+ * @param offset an always incrementing number usually capturing the number of checkpoints taken
+ * by the subtask
+ * @return transactionalId
+ */
+ public static String buildTransactionalId(
+ String transactionalIdPrefix, int subtaskId, long offset) {
+ final StringBuilder sb = new StringBuilder();
+ return sb.append(transactionalIdPrefix)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(subtaskId)
+ .append(TRANSACTIONAL_ID_DELIMITER)
+ .append(offset)
+ .toString();
+ }
+
+ private Consumer<ConsumerRecord<byte[], byte[]>> maybeAddTransaction(
+ Set<String> openTransactions) {
Review comment:
Do you mean instead of using a dedicated method to use an anonymous function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 5916417a1266f4af34d387dd2e61a10673a1a9fe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535)
* 864808edf90294e0fa1386c321ae752a7e076b97 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 5916417a1266f4af34d387dd2e61a10673a1a9fe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890959375
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 483a1a6e982e43b538f13152d41b71144143e444 (Sat Aug 28 13:11:08 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-892702719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af9dd2b341dfe555224f46d3e14a7a83c1b5db04 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294)
* 22e7bb8757f57435a056f4dd720659c5a3207339 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 5916417a1266f4af34d387dd2e61a10673a1a9fe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22e7bb8757f57435a056f4dd720659c5a3207339 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310)
* 0f0ab5854917b30e283dd7195690dd46ee180521 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6287ea41e01d215a3974dca0ae037537a7f3d4bb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397)
* a4a37916fdd2d49a7682b2d90da22e5ed398eae5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-893201715
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684123152
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaWriterState.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class KafkaWriterState {
+ private final int subtaskId;
+ private final long transactionalIdOffset;
+ private final String transactionalIdPrefix;
Review comment:
Okay nice. Do we want to expose it to the user that they may change the prefix or should we communicate that the prefix should remain stable? I'm assuming quite a few edge cases would not work well if a prefix is changed (think of lingering transactions opened before downscaling without recent checkpoint). So I would probably communicate that the prefix is supposed to be stable for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
}, {
"hash" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 864808edf90294e0fa1386c321ae752a7e076b97 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639)
* e8c6ec06f2263f5704ccf597494c43a1bd0e5893 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664)
* 5f636910791eb5ddaa6b9b210c3a4b3230fc60bd UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r684155271
##########
File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/sink/KafkaSinkITCase.java
##########
@@ -0,0 +1,634 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka.sink;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.configuration.StateBackendOptions;
+import org.apache.flink.connector.base.DeliveryGuarantee;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.test.util.TestUtils;
+import org.apache.flink.testutils.junit.SharedObjects;
+import org.apache.flink.testutils.junit.SharedReference;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.CreateTopicsResult;
+import org.apache.kafka.clients.admin.DeleteTopicsResult;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.utility.DockerImageName;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.LongStream;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+/** Tests for using KafkaSink writing to a Kafka cluster. */
+public class KafkaSinkITCase extends TestLogger {
+
+ private static final Logger LOG = LoggerFactory.getLogger(KafkaSinkITCase.class);
+ private static final Slf4jLogConsumer LOG_CONSUMER = new Slf4jLogConsumer(LOG);
+ private static final String INTER_CONTAINER_KAFKA_ALIAS = "kafka";
+ private static final Network NETWORK = Network.newNetwork();
+ private static final int ZK_TIMEOUT_MILLIS = 30000;
+ private static final short TOPIC_REPLICATION_FACTOR = 1;
+ private static final Duration CONSUMER_POLL_DURATION = Duration.ofSeconds(1);
+
+ private String topic;
+ private SharedReference<AtomicLong> emittedRecordsCount;
+ private SharedReference<AtomicLong> emittedRecordsWithCheckpoint;
+ private SharedReference<AtomicBoolean> failed;
+ private SharedReference<AtomicLong> lastCheckpointedRecord;
+
+ @ClassRule
+ public static final KafkaContainer KAFKA_CONTAINER =
+ new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.5.2"))
+ .withEmbeddedZookeeper()
+ .withEnv(
+ ImmutableMap.of(
+ "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR",
+ "1",
+ "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
+ String.valueOf(Duration.ofHours(2).toMillis()),
+ "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR",
+ "1",
+ "KAFKA_MIN_INSYNC_REPLICAS",
+ "1"))
+ .withNetwork(NETWORK)
+ .withLogConsumer(LOG_CONSUMER)
+ .withNetworkAliases(INTER_CONTAINER_KAFKA_ALIAS);
+
+ @Rule public final SharedObjects sharedObjects = SharedObjects.create();
+
+ @Rule public final TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void setUp() throws ExecutionException, InterruptedException, TimeoutException {
+ emittedRecordsCount = sharedObjects.add(new AtomicLong());
+ emittedRecordsWithCheckpoint = sharedObjects.add(new AtomicLong());
+ failed = sharedObjects.add(new AtomicBoolean(false));
+ lastCheckpointedRecord = sharedObjects.add(new AtomicLong(0));
+ topic = UUID.randomUUID().toString();
+ createTestTopic(topic, 1, TOPIC_REPLICATION_FACTOR);
+ }
+
+ @After
+ public void tearDown() throws ExecutionException, InterruptedException, TimeoutException {
+ deleteTestTopic(topic);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithAtLeastOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.AT_LEAST_ONCE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithNoneGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.NONE, emittedRecordsCount);
+ }
+
+ @Test
+ public void testWriteRecordsToKafkaWithExactlyOnceGuarantee() throws Exception {
+ writeRecordsToKafka(DeliveryGuarantee.EXACTLY_ONCE, emittedRecordsWithCheckpoint);
+ }
+
+ @Test
+ public void testRecoveryWithAtLeastOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.AT_LEAST_ONCE,
+ (records) ->
+ assertThat(records, hasItems(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L)));
+ }
+
+ @Test
+ public void testRecoveryWithExactlyOnceGuarantee() throws Exception {
+ testRecoveryWithAssertion(
+ DeliveryGuarantee.EXACTLY_ONCE,
+ (records) ->
+ assertEquals(
+ records,
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList())));
+ }
+
+ @Test
+ public void testAbortTransactionsOfPendingCheckpointsAfterFailure() throws Exception {
+ // Run a first job failing during the async phase of a checkpoint to leave some
+ // lingering transactions
+ final Configuration config = new Configuration();
+ config.setString(StateBackendOptions.STATE_BACKEND, "filesystem");
+ final File checkpointDir = temp.newFolder();
+ config.setString(
+ CheckpointingOptions.CHECKPOINTS_DIRECTORY, checkpointDir.toURI().toString());
+ config.set(
+ ExecutionCheckpointingOptions.EXTERNALIZED_CHECKPOINT,
+ CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
+ config.set(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 2);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(1), config, "firstPrefix");
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ final File completedCheckpoint = TestUtils.getMostRecentCompletedCheckpoint(checkpointDir);
+
+ config.set(SavepointConfigOptions.SAVEPOINT_PATH, completedCheckpoint.toURI().toString());
+
+ // Run a second job which aborts all lingering transactions and new consumer should
+ // immediately see the newly written records
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, "newPrefix");
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ @Test
+ public void testAbortTransactionsAfterScaleInBeforeFirstCheckpoint() throws Exception {
+ // Run a first job opening 5 transactions one per subtask and fail in async checkpoint phase
+ final Configuration config = new Configuration();
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 5);
+ try {
+ executeWithMapper(new FailAsyncCheckpointMapper(0), config, null);
+ } catch (Exception e) {
+ assertThat(
+ e.getCause().getCause().getMessage(),
+ containsString("Exceeded checkpoint tolerable failure"));
+ }
+ assertTrue(deserializeValues(drainAllRecordsFromTopic(topic)).isEmpty());
+
+ // Second job aborts all transactions from previous runs with higher parallelism
+ config.set(CoreOptions.DEFAULT_PARALLELISM, 1);
+ failed.get().set(true);
+ executeWithMapper(
+ new FailingCheckpointMapper(failed, lastCheckpointedRecord), config, null);
+ final List<ConsumerRecord<byte[], byte[]>> collectedRecords =
+ drainAllRecordsFromTopic(topic);
+ assertEquals(
+ deserializeValues(collectedRecords),
+ LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ }
+
+ private void executeWithMapper(
+ MapFunction<Long, Long> mapper,
+ Configuration config,
+ @Nullable String transactionalIdPrefix)
+ throws Exception {
+ final StreamExecutionEnvironment env = new LocalStreamEnvironment(config);
+ env.enableCheckpointing(100L);
+ env.setRestartStrategy(RestartStrategies.noRestart());
+ final DataStreamSource<Long> source = env.fromSequence(1, 10);
+ final DataStream<Long> stream = source.map(mapper);
+ final KafkaSinkBuilder<Long> builder =
+ new KafkaSinkBuilder<Long>()
+ .setKafkaProducerConfig(getKafkaClientConfiguration())
+ .setDeliverGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
+ .setBootstrapServers(KAFKA_CONTAINER.getBootstrapServers())
+ .setRecordSerializer(new RecordSerializer(topic));
+ if (transactionalIdPrefix == null) {
+ transactionalIdPrefix = "kafka-sink";
+ }
+ builder.setTransactionalIdPrefix(transactionalIdPrefix);
+ stream.sinkTo(builder.build());
+ env.execute();
+ checkProducerLeak();
+ }
+
+ private void testRecoveryWithAssertion(
+ DeliveryGuarantee guarantee, java.util.function.Consumer<List<Long>> recordsAssertion)
+ throws Exception {
Review comment:
Thanks for the suggestion, I get the idea. I will try to facilitate it next time or do you want to have the tests refactored?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16676:
URL: https://github.com/apache/flink/pull/16676#issuecomment-890987522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21294",
"triggerID" : "af9dd2b341dfe555224f46d3e14a7a83c1b5db04",
"triggerType" : "PUSH"
}, {
"hash" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21310",
"triggerID" : "22e7bb8757f57435a056f4dd720659c5a3207339",
"triggerType" : "PUSH"
}, {
"hash" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21317",
"triggerID" : "0f0ab5854917b30e283dd7195690dd46ee180521",
"triggerType" : "PUSH"
}, {
"hash" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21397",
"triggerID" : "6287ea41e01d215a3974dca0ae037537a7f3d4bb",
"triggerType" : "PUSH"
}, {
"hash" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21429",
"triggerID" : "a4a37916fdd2d49a7682b2d90da22e5ed398eae5",
"triggerType" : "PUSH"
}, {
"hash" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21464",
"triggerID" : "25fb6f9a84ab137be93a04bfad41c4ab021a3b4b",
"triggerType" : "PUSH"
}, {
"hash" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21494",
"triggerID" : "1e41696e78086dda1970f6e50e3125d0271c056b",
"triggerType" : "PUSH"
}, {
"hash" : "0fd6c57783b7dee619945551d94490442b77af0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fd6c57783b7dee619945551d94490442b77af0a",
"triggerType" : "PUSH"
}, {
"hash" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21520",
"triggerID" : "77d980b81bcd178befe0a1f3762234a2e5b79222",
"triggerType" : "PUSH"
}, {
"hash" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21531",
"triggerID" : "3b5f4809a9c732e5c956b07c21b51faf2fe2f4ff",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"triggerType" : "PUSH"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21574",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "5916417a1266f4af34d387dd2e61a10673a1a9fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21535",
"triggerID" : "893201715",
"triggerType" : "MANUAL"
}, {
"hash" : "864808edf90294e0fa1386c321ae752a7e076b97",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21639",
"triggerID" : "864808edf90294e0fa1386c321ae752a7e076b97",
"triggerType" : "PUSH"
}, {
"hash" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21664",
"triggerID" : "e8c6ec06f2263f5704ccf597494c43a1bd0e5893",
"triggerType" : "PUSH"
}, {
"hash" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f636910791eb5ddaa6b9b210c3a4b3230fc60bd",
"triggerType" : "PUSH"
}, {
"hash" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681",
"triggerID" : "055d0fd83c72f141a1bd2d8a4bc0e30bf637687f",
"triggerType" : "PUSH"
}, {
"hash" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96995fe1f26d7a08e91308409d2c66ec32ec8819",
"triggerType" : "PUSH"
}, {
"hash" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21692",
"triggerID" : "2d1d736dc15e4b96b13a917ecfdd9716d8d48be4",
"triggerType" : "PUSH"
}, {
"hash" : "483a1a6e982e43b538f13152d41b71144143e444",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21701",
"triggerID" : "483a1a6e982e43b538f13152d41b71144143e444",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fd6c57783b7dee619945551d94490442b77af0a UNKNOWN
* 5f636910791eb5ddaa6b9b210c3a4b3230fc60bd UNKNOWN
* 055d0fd83c72f141a1bd2d8a4bc0e30bf637687f Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21681)
* 96995fe1f26d7a08e91308409d2c66ec32ec8819 UNKNOWN
* 2d1d736dc15e4b96b13a917ecfdd9716d8d48be4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21692)
* 483a1a6e982e43b538f13152d41b71144143e444 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21701)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #16676: [FLINK-22902] Port KafkaSink to FLIP-143
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #16676:
URL: https://github.com/apache/flink/pull/16676#discussion_r681803300
##########
File path: flink-connectors/flink-connector-kafka/pom.xml
##########
@@ -78,6 +78,18 @@ under the License.
<version>${kafka.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kafka_${scala.binary.version}</artifactId>
+ <version>${kafka.version}</version>
+ <exclusions>
Review comment:
Unfortunately, I can't exclude all transitive dependencies because the old tests are instantiating a KafkaServer. We can either think of moving the KafkaSink to a different module or keep it as is until we remove all the old tests and can properly exclude the transitive dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org