You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/03 17:26:08 UTC

[GitHub] [beam] kennknowles opened a new issue, #18353: KafkaIO bounded source

kennknowles opened a new issue, #18353:
URL: https://github.com/apache/beam/issues/18353

   KafkaIO could be a useful source for batch applications as well. It could implement a bounded source. The primary question is how the bounds are specified.
   
   One option : Source specifies a time period (say 9am-10am), and KafkaIO fetches appropriate start and end offsets based on time-index in Kafka. This would suite many batch applications that are launched on a scheduled.
   
   Another option is to always read till the end and commit the offsets to Kafka. Handling failures and multiple runs of a task might be complicated.
   
   
   
   Imported from Jira [BEAM-2185](https://issues.apache.org/jira/browse/BEAM-2185). Original Jira may contain additional context.
   Reported by: rangadi.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org