You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "akaltsikis (via GitHub)" <gi...@apache.org> on 2023/12/27 09:58:23 UTC

[PR] KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId [kafka]

akaltsikis opened a new pull request, #15078:
URL: https://github.com/apache/kafka/pull/15078

   KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId
   
   This is expected to respect the timeout that fenceProducers have, in the InitProducerId request.
   
   
   ### Committer Checklist (excluded from commit message)
   - [x] Verify design and implementation 
   - [x] Verify test coverage and CI build status
   - [x] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId [kafka]

Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 closed pull request #15078: KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId
URL: https://github.com/apache/kafka/pull/15078


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId [kafka]

Posted by "edoardocomar (via GitHub)" <gi...@apache.org>.
edoardocomar commented on PR #15078:
URL: https://github.com/apache/kafka/pull/15078#issuecomment-2141607614

   This PR seems stale, please see this alternative https://github.com/apache/kafka/pull/16151


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId [kafka]

Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on code in PR #15078:
URL: https://github.com/apache/kafka/pull/15078#discussion_r1437200184


##########
clients/src/main/java/org/apache/kafka/clients/admin/internals/FenceProducersHandler.java:
##########
@@ -82,9 +86,10 @@ InitProducerIdRequest.Builder buildSingleRequest(int brokerId, CoordinatorKey ke
             .setProducerEpoch(ProducerIdAndEpoch.NONE.epoch)
             .setProducerId(ProducerIdAndEpoch.NONE.producerId)
             .setTransactionalId(key.idValue)
-            // Set transaction timeout to 1 since it's only being initialized to fence out older producers with the same transactional ID,
-            // and shouldn't be used for any actual record writes
-            .setTransactionTimeoutMs(1);
+            // Set transaction timeout to the equivalent as the fenceProducers request since it's only being initialized to fence out older producers with the same transactional ID,
+            // and shouldn't be used for any actual record writes. This has been changed to match the fenceProducers request timeout from one as some brokers may be slower than expected
+            // and we need a safe timeout that allows the transaction init to finish.
+            .setTransactionTimeoutMs(this.options.timeoutMs());

Review Comment:
   The timeoutMs is nullable, and is always null when using the default FenceProducersOptions.
   When it is null we should use the KafkaAdminClient#defaultApiTimeoutMs, similar to deleteRecords or calcDeadlineMs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId [kafka]

Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on PR #15078:
URL: https://github.com/apache/kafka/pull/15078#issuecomment-2150760941

   Closing in favor of #16151. Thanks @akaltsikis, I hope you have more time to work on Kafka in the future :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId [kafka]

Posted by "akaltsikis (via GitHub)" <gi...@apache.org>.
akaltsikis commented on code in PR #15078:
URL: https://github.com/apache/kafka/pull/15078#discussion_r1443743036


##########
clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java:
##########
@@ -4394,7 +4394,10 @@ public ListTransactionsResult listTransactions(ListTransactionsOptions options)
     public FenceProducersResult fenceProducers(Collection<String> transactionalIds, FenceProducersOptions options) {
         AdminApiFuture.SimpleAdminApiFuture<CoordinatorKey, ProducerIdAndEpoch> future =
             FenceProducersHandler.newFuture(transactionalIds);
-        FenceProducersHandler handler = new FenceProducersHandler(logContext);
+        if (options.timeoutMs() == null) {
+            options.timeoutMs(defaultApiTimeoutMs);

Review Comment:
   Thanks for the good comment.
   
   The main reason i mutated the options was to pass the same timeout value in both the `FenceProducersHandler` and in the `invokeDriver`.
   
   I just noticed that this is already true as the [calcDeadlineMs](https://github.com/apache/kafka/blob/8d1a1dd7f2655e411e48ef5043b1385fd9804991/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L469-L473) that is used by the `invokeDriver` is already returning the `defaultApiTimeoutMs` in the case of null timeout.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId [kafka]

Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on code in PR #15078:
URL: https://github.com/apache/kafka/pull/15078#discussion_r1438351308


##########
clients/src/test/java/org/apache/kafka/clients/admin/KafkaAdminClientTest.java:
##########
@@ -7031,6 +7031,7 @@ public void testFenceProducers() throws Exception {
         try (AdminClientUnitTestEnv env = mockClientEnv()) {
             String transactionalId = "copyCat";
             Node transactionCoordinator = env.cluster().nodes().iterator().next();
+            final FenceProducersOptions options = new FenceProducersOptions().timeoutMs(10000);

Review Comment:
   This doesn't test the null-default case which is going to be more common, and doesn't seem necessary for the correctness of the test either.
   
   I think instead, we should change the `request -> request instanceof InitProducerIdRequest` to assert that the increased timeout is used, instead of the 1ms timeout. And maybe refactor/parameterize the test to try a default-options and options with an explicit timeout, if we want to test both branches.



##########
clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java:
##########
@@ -4394,7 +4394,10 @@ public ListTransactionsResult listTransactions(ListTransactionsOptions options)
     public FenceProducersResult fenceProducers(Collection<String> transactionalIds, FenceProducersOptions options) {
         AdminApiFuture.SimpleAdminApiFuture<CoordinatorKey, ProducerIdAndEpoch> future =
             FenceProducersHandler.newFuture(transactionalIds);
-        FenceProducersHandler handler = new FenceProducersHandler(logContext);
+        if (options.timeoutMs() == null) {
+            options.timeoutMs(defaultApiTimeoutMs);

Review Comment:
   I think it's best if we don't mutate the options if the user passed it in. If someone were to log the options somewhere else, they would see our injected default instead of the fact that no value was provided.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15078:
URL: https://github.com/apache/kafka/pull/15078#issuecomment-2040935039

   This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has  merge conflicts, please update it with the latest from trunk (or appropriate release branch) <p> If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16047: Leverage the fenceProducers timeout in the InitProducerId [kafka]

Posted by "akaltsikis (via GitHub)" <gi...@apache.org>.
akaltsikis commented on code in PR #15078:
URL: https://github.com/apache/kafka/pull/15078#discussion_r1437810684


##########
clients/src/main/java/org/apache/kafka/clients/admin/internals/FenceProducersHandler.java:
##########
@@ -82,9 +86,10 @@ InitProducerIdRequest.Builder buildSingleRequest(int brokerId, CoordinatorKey ke
             .setProducerEpoch(ProducerIdAndEpoch.NONE.epoch)
             .setProducerId(ProducerIdAndEpoch.NONE.producerId)
             .setTransactionalId(key.idValue)
-            // Set transaction timeout to 1 since it's only being initialized to fence out older producers with the same transactional ID,
-            // and shouldn't be used for any actual record writes
-            .setTransactionTimeoutMs(1);
+            // Set transaction timeout to the equivalent as the fenceProducers request since it's only being initialized to fence out older producers with the same transactional ID,
+            // and shouldn't be used for any actual record writes. This has been changed to match the fenceProducers request timeout from one as some brokers may be slower than expected
+            // and we need a safe timeout that allows the transaction init to finish.
+            .setTransactionTimeoutMs(this.options.timeoutMs());

Review Comment:
   Can you please recheck @gharris1727 ? 🙏



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org