You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/07/30 17:52:24 UTC
[GitHub] [kafka] guozhangwang opened a new pull request #11151: MINOR: Should commit a task if the consumer position advanced as well
guozhangwang opened a new pull request #11151:
URL: https://github.com/apache/kafka/pull/11151
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a change in pull request #11151: MINOR: Should commit a task if the consumer position advanced as well
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on a change in pull request #11151:
URL: https://github.com/apache/kafka/pull/11151#discussion_r682935981
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java
##########
@@ -1158,7 +1158,38 @@ public String toString(final String indent) {
@Override
public boolean commitNeeded() {
- return commitNeeded;
+ // we need to do an extra check if the flag was false, that
+ // if the consumer position has been updated; this is because
+ // there may be non data records such as control markers bypassed
+ if (commitNeeded) {
+ return true;
+ } else {
+ for (final Map.Entry<TopicPartition, Long> entry : consumedOffsets.entrySet()) {
+ final TopicPartition partition = entry.getKey();
+ try {
+ final long offset = mainConsumer.position(partition);
+
+ // note the position in consumer is the "next" record to fetch,
+ // so it should be larger than the consumed offset by 1; if it is
+ // more than 1 it means there are skipped offsets
Review comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #11151: MINOR: Should commit a task if the consumer position advanced as well
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #11151:
URL: https://github.com/apache/kafka/pull/11151#discussion_r680997728
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java
##########
@@ -1158,7 +1158,38 @@ public String toString(final String indent) {
@Override
public boolean commitNeeded() {
- return commitNeeded;
+ // we need to do an extra check if the flag was false, that
+ // if the consumer position has been updated; this is because
+ // there may be non data records such as control markers bypassed
+ if (commitNeeded) {
+ return true;
+ } else {
+ for (final Map.Entry<TopicPartition, Long> entry : consumedOffsets.entrySet()) {
+ final TopicPartition partition = entry.getKey();
+ try {
+ final long offset = mainConsumer.position(partition);
+
+ // note the position in consumer is the "next" record to fetch,
+ // so it should be larger than the consumed offset by 1; if it is
+ // more than 1 it means there are skipped offsets
Review comment:
"skipped offsets" sounds scary. How about "control records, which the consumer skips over silently"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a change in pull request #11151: MINOR: Should commit a task if the consumer position advanced as well
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on a change in pull request #11151:
URL: https://github.com/apache/kafka/pull/11151#discussion_r682935981
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java
##########
@@ -1158,7 +1158,38 @@ public String toString(final String indent) {
@Override
public boolean commitNeeded() {
- return commitNeeded;
+ // we need to do an extra check if the flag was false, that
+ // if the consumer position has been updated; this is because
+ // there may be non data records such as control markers bypassed
+ if (commitNeeded) {
+ return true;
+ } else {
+ for (final Map.Entry<TopicPartition, Long> entry : consumedOffsets.entrySet()) {
+ final TopicPartition partition = entry.getKey();
+ try {
+ final long offset = mainConsumer.position(partition);
+
+ // note the position in consumer is the "next" record to fetch,
+ // so it should be larger than the consumed offset by 1; if it is
+ // more than 1 it means there are skipped offsets
Review comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang merged pull request #11151: MINOR: Should commit a task if the consumer position advanced as well
Posted by GitBox <gi...@apache.org>.
guozhangwang merged pull request #11151:
URL: https://github.com/apache/kafka/pull/11151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on pull request #11151: MINOR: Should commit a task if the consumer position advanced as well
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on pull request #11151:
URL: https://github.com/apache/kafka/pull/11151#issuecomment-890057040
@vvcephei @AlanConfluent this is not a ready to merge PR, but just demonstrating what I meant to fix the issue.
I will add the test coverage after you've agreed this is the right fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org