hello, I'm user in kafka having a question about 'commit'

[김세곤 <sg...@naver.com>]

 
hello, I'm user in kafka
 
I want find commit method from consumer to zookeeper, but can't find it
I want to make structure like that,
 
hadoop-consumer: 1. it get message
2. it write message to hadoop hdfs
3. it have to get message again from what it read ,when it is fault and recover.
( for example,  consumer was reading message
suddenly it was fault. during fault time, message is producing.
after recovering, consumer have to read message from what it didn't read offset, not from first)
 
 
problem is 3.
 
I read about offset, written that
after consumer read message, it commit to zookeeper. using that offset ( ex, getbeforeoffset() ), can read message again.
But, using this method offer that reading from first(offset-1) to last. And if use not it, consumer can't get messages produced during fault
 
So, I want find any source or method to satisfy my structure, is there any thing to help me?
 
 
thank you for reading
please give some help 
 
 

Re: hello, I'm user in kafka having a question about 'commit'

[Jun Rao <ju...@gmail.com>]

If you are using the high level consumer, there is an commitOffsets() api.
If you are using SimpleConsumer, you are on your own for offset management.

The getOffsetBefore api is not for getting the consumer offsets, but for
getting the offset in the log before a particular time.

Thanks,

Jun


On Tue, Jan 28, 2014 at 10:51 PM, ê¹ EURO ì"¸ê³¤ <sg...@naver.com> wrote:

>
> hello, I'm user in kafka
>
> I want find commit method from consumer to zookeeper, but can't find it
> I want to make structure like that,
>
> hadoop-consumer: 1. it get message
> 2. it write message to hadoop hdfs
> 3. it have to get message again from what it read ,when it is fault and
> recover.
> ( for example,  consumer was reading message
> suddenly it was fault. during fault time, message is producing.
> after recovering, consumer have to read message from what it didn't read
> offset, not from first)
>
>
> problem is 3.
>
> I read about offset, written that
> after consumer read message, it commit to zookeeper. using that offset (
> ex, getbeforeoffset() ), can read message again.
> But, using this method offer that reading from first(offset-1) to last.
> And if use not it, consumer can't get messages produced during fault
>
> So, I want find any source or method to satisfy my structure, is there any
> thing to help me?
>
>
> thank you for reading
> please give some help
>
>
>