You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mjsax (via GitHub)" <gi...@apache.org> on 2023/04/10 22:31:06 UTC

[GitHub] [kafka] mjsax opened a new pull request, #13534: KAFKA-14054: Handle TimeoutException gracefully

mjsax opened a new pull request, #13534:
URL: https://github.com/apache/kafka/pull/13534

   We incorrectly assumed, that `consumer.position()` should always be served by the consumer locally set position.
   
   However, within `commitNeeded()` we check if first `if(commitNeeded)` and thus go into the else only if we have not processed data (otherwise, `commitNeeded` would be true). For this reason, we actually don't know if the consumer has a valid position or not.
   
   We should just swallow a timeout if the consumer cannot get the position from the broker, and try the next partition. If any position advances, we can return true, and if we timeout for all partitions we can return false.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] miccab commented on a diff in pull request #13534: KAFKA-14054: Handle TimeoutException gracefully

Posted by "miccab (via GitHub)" <gi...@apache.org>.
miccab commented on code in PR #13534:
URL: https://github.com/apache/kafka/pull/13534#discussion_r1165085371


##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -1195,13 +1195,8 @@ public boolean commitNeeded() {
                         commitNeeded = true;
                         entry.setValue(offset - 1);
                     }
-                } catch (final TimeoutException error) {
-                    // the `consumer.position()` call should never block, because we know that we did process data
-                    // for the requested partition and thus the consumer should have a valid local position
-                    // that it can return immediately
-
-                    // hence, a `TimeoutException` indicates a bug and thus we rethrow it as fatal `IllegalStateException`
-                    throw new IllegalStateException(error);
+                } catch (final TimeoutException swallow) {
+                    log.debug("Could not get consumer position for partition {} due to: {}", partition, swallow);

Review Comment:
   Makes sense. Thanks for detailed explanation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #13534: KAFKA-14054: Handle TimeoutException gracefully

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #13534:
URL: https://github.com/apache/kafka/pull/13534#discussion_r1164442798


##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -1195,13 +1195,8 @@ public boolean commitNeeded() {
                         commitNeeded = true;
                         entry.setValue(offset - 1);
                     }
-                } catch (final TimeoutException error) {
-                    // the `consumer.position()` call should never block, because we know that we did process data
-                    // for the requested partition and thus the consumer should have a valid local position
-                    // that it can return immediately
-
-                    // hence, a `TimeoutException` indicates a bug and thus we rethrow it as fatal `IllegalStateException`
-                    throw new IllegalStateException(error);
+                } catch (final TimeoutException swallow) {
+                    log.debug("Could not get consumer position for partition {} due to: {}", partition, swallow);

Review Comment:
   Good question.
   
   I think both cases are different:
   
   1. In `commitNeeded()`, we call `consumer.position()` if the flag `commitNeeded == false`, ie, we know that we did not process any record, and thus, we cannot be sure that the consumer has a locally cached valid position. (Note, that we introduce this code in AK 3.1.0 release, and it seems it was more or less copied from `findOffset` which is older.)
   
   2. `findOffset` on the other hand, is only called for partitions which have the `commitNeeded` flag set as `true` (cf `committableOffsetsAndMetadata()` where we call `findOffset`). This code was already added in 2.6 release. The ticket is also reported only for `commitNeeded` case above, indicating that this older code is correct, while the bug was only introduced in 3.1 when the code might have be copied incorrectly.
   
   Does this make sense? Or do I miss anything?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on pull request #13534: KAFKA-14054: Handle TimeoutException gracefully

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13534:
URL: https://github.com/apache/kafka/pull/13534#issuecomment-1509012485

   Merged to `trunk` and cherry-picked to `3.5` branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] guozhangwang commented on a diff in pull request #13534: KAFKA-14054: Handle TimeoutException gracefully

Posted by "guozhangwang (via GitHub)" <gi...@apache.org>.
guozhangwang commented on code in PR #13534:
URL: https://github.com/apache/kafka/pull/13534#discussion_r1163035279


##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -1195,13 +1195,8 @@ public boolean commitNeeded() {
                         commitNeeded = true;
                         entry.setValue(offset - 1);
                     }
-                } catch (final TimeoutException error) {
-                    // the `consumer.position()` call should never block, because we know that we did process data
-                    // for the requested partition and thus the consumer should have a valid local position
-                    // that it can return immediately
-
-                    // hence, a `TimeoutException` indicates a bug and thus we rethrow it as fatal `IllegalStateException`
-                    throw new IllegalStateException(error);
+                } catch (final TimeoutException swallow) {
+                    log.debug("Could not get consumer position for partition {} due to: {}", partition, swallow);

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax merged pull request #13534: KAFKA-14054: Handle TimeoutException gracefully

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #13534:
URL: https://github.com/apache/kafka/pull/13534


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] vvcephei commented on a diff in pull request #13534: KAFKA-14054: Handle TimeoutException gracefully

Posted by "vvcephei (via GitHub)" <gi...@apache.org>.
vvcephei commented on code in PR #13534:
URL: https://github.com/apache/kafka/pull/13534#discussion_r1163018118


##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -1195,13 +1195,8 @@ public boolean commitNeeded() {
                         commitNeeded = true;
                         entry.setValue(offset - 1);
                     }
-                } catch (final TimeoutException error) {
-                    // the `consumer.position()` call should never block, because we know that we did process data
-                    // for the requested partition and thus the consumer should have a valid local position
-                    // that it can return immediately
-
-                    // hence, a `TimeoutException` indicates a bug and thus we rethrow it as fatal `IllegalStateException`
-                    throw new IllegalStateException(error);
+                } catch (final TimeoutException swallow) {
+                    log.debug("Could not get consumer position for partition {} due to: {}", partition, swallow);

Review Comment:
   This won't print the exception. It's using https://www.javadoc.io/static/org.slf4j/slf4j-api/1.7.30/org/slf4j/Logger.html#debug-java.lang.String-java.lang.Object-java.lang.Object- , but the intent is probably to use https://www.javadoc.io/static/org.slf4j/slf4j-api/1.7.30/org/slf4j/Logger.html#debug-java.lang.String-java.lang.Throwable- , which requires a fully constructed string as the first arg.
   
   ```suggestion
                       log.debug(String.format("Could not get consumer position for partition %d", partition), swallow);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] miccab commented on a diff in pull request #13534: KAFKA-14054: Handle TimeoutException gracefully

Posted by "miccab (via GitHub)" <gi...@apache.org>.
miccab commented on code in PR #13534:
URL: https://github.com/apache/kafka/pull/13534#discussion_r1163759932


##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -1195,13 +1195,8 @@ public boolean commitNeeded() {
                         commitNeeded = true;
                         entry.setValue(offset - 1);
                     }
-                } catch (final TimeoutException error) {
-                    // the `consumer.position()` call should never block, because we know that we did process data
-                    // for the requested partition and thus the consumer should have a valid local position
-                    // that it can return immediately
-
-                    // hence, a `TimeoutException` indicates a bug and thus we rethrow it as fatal `IllegalStateException`
-                    throw new IllegalStateException(error);
+                } catch (final TimeoutException swallow) {
+                    log.debug("Could not get consumer position for partition {} due to: {}", partition, swallow);

Review Comment:
   should there not be the same change inside this method:
   private Long findOffset(final TopicPartition partition) {



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org