You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "C0urante (via GitHub)" <gi...@apache.org> on 2024/01/31 17:56:07 UTC

[PR] KAFKA-15524, KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

C0urante opened a new pull request, #15302:
URL: https://github.com/apache/kafka/pull/15302

   [Jira 1](https://issues.apache.org/jira/browse/KAFKA-15524), [Jira 2](https://issues.apache.org/jira/browse/KAFKA-15917)
   
   In some rare cases, our integration tests observe that tasks are up and running for our sink connectors, and issue a subsequent request to stop the connector, before the consumers of these tasks have had a chance to form/join a group. When this happens, requests to alter/reset the offsets for the connector actually succeed, since the group either doesn't exist, or is completely empty.
   
   As a quick fix, we can wait for offsets to be committed for the connector before stopping it and attempting to modify its offsets. This goes a bit further than necessary (we should only have to wait for at least one task's consumer to have formed/joined a group), but off the top of my head it was the cleanest and briefest way to guarantee that a group had been formed. It also makes a little more sense in the context of the test, since there's not much use in modifying connector offsets when none exist yet.
   
   Opening as a draft; will wait for CI to complete before marking ready for review.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante merged PR #15302:
URL: https://github.com/apache/kafka/pull/15302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on code in PR #15302:
URL: https://github.com/apache/kafka/pull/15302#discussion_r1663319861


##########
connect/runtime/src/test/java/org/apache/kafka/connect/integration/OffsetsApiIntegrationTest.java:
##########
@@ -801,6 +806,11 @@ public void testResetSinkConnectorOffsetsZombieSinkTasks() throws Exception {
         connect.assertions().assertConnectorAndAtLeastNumTasksAreRunning(connectorName, 1,
                 "Connector tasks did not start in time.");
 
+        // Make sure the tasks' consumers have had a chance to actually form a group
+        // (otherwise, the reset request will succeed because there won't be any active consumers)
+        verifyExpectedSinkConnectorOffsets(connectorName, topic, 1, NUM_RECORDS_PER_PARTITION,

Review Comment:
   Great point, I've removed mention of KAFKA-15224 from this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #15302:
URL: https://github.com/apache/kafka/pull/15302#issuecomment-2206277714

   Thanks Yash!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on code in PR #15302:
URL: https://github.com/apache/kafka/pull/15302#discussion_r1663319861


##########
connect/runtime/src/test/java/org/apache/kafka/connect/integration/OffsetsApiIntegrationTest.java:
##########
@@ -801,6 +806,11 @@ public void testResetSinkConnectorOffsetsZombieSinkTasks() throws Exception {
         connect.assertions().assertConnectorAndAtLeastNumTasksAreRunning(connectorName, 1,
                 "Connector tasks did not start in time.");
 
+        // Make sure the tasks' consumers have had a chance to actually form a group
+        // (otherwise, the reset request will succeed because there won't be any active consumers)
+        verifyExpectedSinkConnectorOffsets(connectorName, topic, 1, NUM_RECORDS_PER_PARTITION,

Review Comment:
   Great point, I've removed mention of KAFKA-15524 from this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15524, KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #15302:
URL: https://github.com/apache/kafka/pull/15302#discussion_r1475872307


##########
connect/runtime/src/test/java/org/apache/kafka/connect/integration/OffsetsApiIntegrationTest.java:
##########
@@ -801,6 +806,11 @@ public void testResetSinkConnectorOffsetsZombieSinkTasks() throws Exception {
         connect.assertions().assertConnectorAndAtLeastNumTasksAreRunning(connectorName, 1,
                 "Connector tasks did not start in time.");
 
+        // Make sure the tasks' consumers have had a chance to actually form a group
+        // (otherwise, the reset request will succeed because there won't be any active consumers)
+        verifyExpectedSinkConnectorOffsets(connectorName, topic, 1, NUM_RECORDS_PER_PARTITION,

Review Comment:
   Looks like this test was already verifying this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15524, KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #15302:
URL: https://github.com/apache/kafka/pull/15302#issuecomment-1921719937

   @yashmayya @gharris1727 If either of you have a moment, here's another flaky test fix. Would appreciate a look!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #15302:
URL: https://github.com/apache/kafka/pull/15302#discussion_r1663466038


##########
connect/runtime/src/test/java/org/apache/kafka/connect/integration/OffsetsApiIntegrationTest.java:
##########
@@ -976,27 +988,53 @@ private String modifySinkConnectorOffsetsWithRetry(ConnectorOffsets offsetsToAlt
      *
      * @param connectorName the name of the sink connector whose offsets are to be verified
      * @param expectedTopic the name of the Kafka topic that the sink connector is consuming from
-     * @param expectedPartitions the number of partitions that exist for the Kafka topic
+     * @param expectedPartitions the number of partitions that exist for the Kafka<<< topic

Review Comment:
   Stray change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "C0urante (via GitHub)" <gi...@apache.org>.
C0urante commented on PR #15302:
URL: https://github.com/apache/kafka/pull/15302#issuecomment-2204763797

   @gharris1727 sorry for the delay. You're correct that this PR did not actually address KAFKA-15224, and I'm honestly not sure how I missed that the first time around. I've updated the description to only mention KAFKA-15917, added some useful info to assertion messages when sink connectors fail to catch up to the expected committed offsets in these tests, and rebased on the latest trunk.
   
   Would you mind giving this another pass when you have a moment? Currently this test is the flakiest of all Connect integration tests, failing nearly 10% of the time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15524, KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on code in PR #15302:
URL: https://github.com/apache/kafka/pull/15302#discussion_r1478735608


##########
connect/runtime/src/test/java/org/apache/kafka/connect/integration/OffsetsApiIntegrationTest.java:
##########
@@ -801,6 +806,11 @@ public void testResetSinkConnectorOffsetsZombieSinkTasks() throws Exception {
         connect.assertions().assertConnectorAndAtLeastNumTasksAreRunning(connectorName, 1,
                 "Connector tasks did not start in time.");
 
+        // Make sure the tasks' consumers have had a chance to actually form a group
+        // (otherwise, the reset request will succeed because there won't be any active consumers)
+        verifyExpectedSinkConnectorOffsets(connectorName, topic, 1, NUM_RECORDS_PER_PARTITION,

Review Comment:
   And since this commit no longer changes the behavior of the Reset test, does that mean that KAFKA-15224 is not addressed in this commit? Is that flaky failure still possible?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15524, KAFKA-15917: Wait for zombie sink tasks' consumers to commit offsets before trying to modify their offsets in integration tests [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15302:
URL: https://github.com/apache/kafka/pull/15302#issuecomment-2095133015

   This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has  merge conflicts, please update it with the latest from trunk (or appropriate release branch) <p> If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org