[GitHub] [beam] chamikaramj commented on a diff in pull request #22261: Add comments and logs to warn about Kafka sdf not properly restarting

[GitBox <gi...@apache.org>]

chamikaramj commented on code in PR #22261:
URL: https://github.com/apache/beam/pull/22261#discussion_r920226998


##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java:
##########
@@ -543,6 +543,11 @@
  * For any significant significant updates to this I/O connector, please consider involving
  * corresponding code reviewers mentioned <a
  * href="https://github.com/apache/beam/blob/master/sdks/java/io/kafka/OWNERS">here</a>.
+ *
+ * <h1>Reading from Kafka SDF is currently broken, as re-starting the pipeline will cause the
+ * consumer to start from scratch. See <a
+ * href="https://github.com/apache/beam/issues/21730">this</a>. Current workaround is to use
+ * --experimental_option=use_deprecated_read to use the Unbounded implementation</h1>

Review Comment:
   I don't think this will work for Dataflow Runner v2 Java pipelines. Can you try ?
   
   "use_unbounded_sdf_wrapper" should work but I only tried it for x-lang.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org