You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by 郑洁锋 <zj...@hotmail.com> on 2019/12/02 02:54:49 UTC

回复: How to store&read offset with kafkaio

Can Apache Beam start from a specified offset?

________________________________
zjfplayer@hotmail.com

发件人： zjfplayer@hotmail.com<ma...@hotmail.com>
发送时间： 2019-11-28 11:32
收件人： user<ma...@beam.apache.org>
主题： How to store&read offset with kafkaio
Hi，
       I want to store the offset in Oracle/Mysql, and then every time I start the real-time streaming task, I want this task to start consuming from the offset stored in Oracle/Mysql.
       I haven't seen how to manually store the offset and specify the offset to start consuming after reading the documentation.

       Can anyone help me
________________________________
zjfplayer@hotmail.com

Re: How to store&read offset with kafkaio

Posted by Alexey Romanenko <ar...@gmail.com>.

I don’t believe that you can do that but there are several other options, like using “withStartReadTime()” or specific group.id
Have you seen this answer https://stackoverflow.com/a/50766439/10687325 ? 

> On 2 Dec 2019, at 03:54, 郑 洁锋 <zj...@hotmail.com> wrote:
> 
> Can Apache Beam start from a specified offset?
> 
> zjfplayer@hotmail.com <ma...@hotmail.com>
>  
> 发件人： zjfplayer@hotmail.com <ma...@hotmail.com>
> 发送时间： 2019-11-28 11:32
> 收件人： user <ma...@beam.apache.org>
> 主题： How to store&read offset with kafkaio
> Hi，
>        I want to store the offset in Oracle/Mysql, and then every time I start the real-time streaming task, I want this task to start consuming from the offset stored in Oracle/Mysql.
>        I haven't seen how to manually store the offset and specify the offset to start consuming after reading the documentation.
> 
>        Can anyone help me
> zjfplayer@hotmail.com <ma...@hotmail.com>