You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 14:47:22 UTC

[GitHub] [beam] damccorm opened a new issue, #20034: Store&Read offset with KafkaIO

damccorm opened a new issue, #20034:
URL: https://github.com/apache/beam/issues/20034

   When creating a Pipeline through a KafkaIO object, I want to be able to specify the starting offset of consumption, and when traversing the message later, I can get the offset of the current message for storage in a relational database / NoSQL.
    
   This feature is used to implement the exactly-once semantics of spark streaming consumption.
    
   In the "Your own data store" section of the following url content, you can find how to achieve exactly-once semantics with spark streaming:
   [http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html](http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html)
   
   Imported from Jira [BEAM-8934](https://issues.apache.org/jira/browse/BEAM-8934). Original Jira may contain additional context.
   Reported by: zjfplayer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org