[GitHub] jsun98 commented on a change in pull request #6496: Support kafka transactional topics (#5404)

[GitBox <gi...@apache.org>]

jsun98 commented on a change in pull request #6496: Support kafka transactional topics (#5404)
URL: https://github.com/apache/incubator-druid/pull/6496#discussion_r253278888
 
 

 ##########
 File path: extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/IncrementalPublishingKafkaIndexTaskRunner.java
 ##########
 @@ -576,15 +561,19 @@ public void onFailure(Throwable t)
 
               nextOffsets.put(record.partition(), record.offset() + 1);
             }
-
-            if (nextOffsets.get(record.partition()).equals(endOffsets.get(record.partition()))
+            if (nextOffsets.get(record.partition()) >= endOffsets.get(record.partition())
                 && assignment.remove(record.partition())) {
               log.info("Finished reading topic[%s], partition[%,d].", record.topic(), record.partition());
               KafkaIndexTask.assignPartitions(consumer, topic, assignment);
               stillReading = !assignment.isEmpty();
             }
           }
-
+          if (nextOffsets.get(currentPartition) != null
 
 Review comment:
   Sorry for the late reply. If I understand correctly, the line `nextOffsets.put(currentPartition, endOffsets.get(currentPartition));` is supposed to update the `nextOffsets` of the partitions with polled offsets that have exceeded the partition's `endOffsets`. But what if, in the list of polled `records`, 2 or more partitions have offsets that exceed their respective endOffsets. Then since `nextOffsets.put(currentPartition, endOffsets.get(currentPartition));` is outside the loop, it would only update the `nextOffsets` of 1 such partition. I remember talking to @jihoonson about this, he may know more.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org