Re: [PR] [KafkaIO] Make ReadFromKafkaDoFn restriction trackers unsplittable [beam]

["scwhittle (via GitHub)" <gi...@apache.org>]

scwhittle commented on code in PR #36935:
URL: https://github.com/apache/beam/pull/36935#discussion_r2576562216


##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFn.java:
##########
@@ -108,6 +109,14 @@
  *
  * <h4>Splitting</h4>
  *
+ * <p>Consumer group members must not consume from the same {@link TopicPartition} simultaneously
+ * when {@code enable.auto.commit} is set. Doing so may arbitrarily overwrite a consumer group's
+ * committed offset for a {@link TopicPartition}. Restriction trackers for a {@link
+ * KafkaSourceDescriptor} are wrapped as {@link UnsplittableRestrictionTracker<OffsetRange, Long>}
+ * and will only return a non-null {@link org.apache.beam.sdk.transforms.splittabledofn.SplitResult}
+ * for a checkpoint. This ensures consistent behavior when {@code enable.auto.commit} is set and

Review Comment:
   nit: "ensures consistent" is too strong I think?  There may still be parallel scheduling on different VMs during scaling etc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org