You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by tao li <tu...@gmail.com> on 2012/06/19 07:41:43 UTC
[kafka-users] how to make Transaction get with the consumer
High-level API
hi, all
In my project, we want to consume the msg in kafka in the transaction, so,
when the transaction rollback, we need the kafka offset back to the
point transaction begin, I use the consumer Low-level API make it, it's too
complex, how to make Transaction get with the consumer High-level API if
is possible.
Best regards,
litao
Re: [kafka-users] how to make Transaction get with the consumer
High-level API
Posted by Jun Rao <ju...@gmail.com>.
Actually, in 0.7.1, the only metadata that we associate with each iterated
message is topic. There are are couple of reasons why we didn't expose
offset: (1) in 0.7.1, partition is local to a broker. So, for each offset,
we have to return a (brokerid, partition id, offset) triple, which
is inconvenient. (2) Our consumer api doesn't support setting the initial
offset for a partition at startup.
So, for now, if you really need to know/set the offset, you have to use the
low-level consumer.
Thanks,
Jun
On Tue, Jun 19, 2012 at 9:22 AM, Jay Kreps <ja...@gmail.com> wrote:
> Jun, I think what he is saying is he wants to keep the offset in the
> database so he can commit the data and offset transactionally.
>
> Tao, in the 0.7 release the offset is not exposed in the high-level API.
> However in 0.7.1 which is currently in RC we do expose this. My
> recommendation would be to give that a try.
>
> -Jay
>
> On Tue, Jun 19, 2012 at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Tao,
> >
> > You can use manual offset commit in the high level consumer. Basically,
> > call offset commit only after the data has been successfully processed.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jun 18, 2012 at 10:41 PM, tao li <tu...@gmail.com>
> wrote:
> >
> > > hi, all
> > >
> > > In my project, we want to consume the msg in kafka in the transaction,
> > so,
> > > when the transaction rollback, we need the kafka offset back to the
> > > point transaction begin, I use the consumer Low-level API make it, it's
> > too
> > > complex, how to make Transaction get with the consumer High-level API
> if
> > > is possible.
> > >
> > > Best regards,
> > >
> > > litao
> > >
> >
>
Re: [kafka-users] how to make Transaction get with the consumer
High-level API
Posted by Jay Kreps <ja...@gmail.com>.
Ack, my mistake...
-Jay
On Tue, Jun 19, 2012 at 10:17 AM, Joel Koshy <jj...@gmail.com> wrote:
> I don't think we expose any offset information in the high-level API in
> 0.7.1 right? We have MessageAndMetadata but the only metadata exposed is
> the topic the message comes from.
>
> Thanks,
>
> Joel
>
> On Tue, Jun 19, 2012 at 9:22 AM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Jun, I think what he is saying is he wants to keep the offset in the
> > database so he can commit the data and offset transactionally.
> >
> > Tao, in the 0.7 release the offset is not exposed in the high-level API.
> > However in 0.7.1 which is currently in RC we do expose this. My
> > recommendation would be to give that a try.
> >
> > -Jay
> >
> > On Tue, Jun 19, 2012 at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Tao,
> > >
> > > You can use manual offset commit in the high level consumer. Basically,
> > > call offset commit only after the data has been successfully processed.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jun 18, 2012 at 10:41 PM, tao li <tu...@gmail.com>
> > wrote:
> > >
> > > > hi, all
> > > >
> > > > In my project, we want to consume the msg in kafka in the
> transaction,
> > > so,
> > > > when the transaction rollback, we need the kafka offset back to the
> > > > point transaction begin, I use the consumer Low-level API make it,
> it's
> > > too
> > > > complex, how to make Transaction get with the consumer High-level
> API
> > if
> > > > is possible.
> > > >
> > > > Best regards,
> > > >
> > > > litao
> > > >
> > >
> >
>
Re: [kafka-users] how to make Transaction get with the consumer
High-level API
Posted by Joel Koshy <jj...@gmail.com>.
I don't think we expose any offset information in the high-level API in
0.7.1 right? We have MessageAndMetadata but the only metadata exposed is
the topic the message comes from.
Thanks,
Joel
On Tue, Jun 19, 2012 at 9:22 AM, Jay Kreps <ja...@gmail.com> wrote:
> Jun, I think what he is saying is he wants to keep the offset in the
> database so he can commit the data and offset transactionally.
>
> Tao, in the 0.7 release the offset is not exposed in the high-level API.
> However in 0.7.1 which is currently in RC we do expose this. My
> recommendation would be to give that a try.
>
> -Jay
>
> On Tue, Jun 19, 2012 at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Tao,
> >
> > You can use manual offset commit in the high level consumer. Basically,
> > call offset commit only after the data has been successfully processed.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jun 18, 2012 at 10:41 PM, tao li <tu...@gmail.com>
> wrote:
> >
> > > hi, all
> > >
> > > In my project, we want to consume the msg in kafka in the transaction,
> > so,
> > > when the transaction rollback, we need the kafka offset back to the
> > > point transaction begin, I use the consumer Low-level API make it, it's
> > too
> > > complex, how to make Transaction get with the consumer High-level API
> if
> > > is possible.
> > >
> > > Best regards,
> > >
> > > litao
> > >
> >
>
Re: [kafka-users] how to make Transaction get with the consumer
High-level API
Posted by Jay Kreps <ja...@gmail.com>.
Jun, I think what he is saying is he wants to keep the offset in the
database so he can commit the data and offset transactionally.
Tao, in the 0.7 release the offset is not exposed in the high-level API.
However in 0.7.1 which is currently in RC we do expose this. My
recommendation would be to give that a try.
-Jay
On Tue, Jun 19, 2012 at 9:16 AM, Jun Rao <ju...@gmail.com> wrote:
> Tao,
>
> You can use manual offset commit in the high level consumer. Basically,
> call offset commit only after the data has been successfully processed.
>
> Thanks,
>
> Jun
>
> On Mon, Jun 18, 2012 at 10:41 PM, tao li <tu...@gmail.com> wrote:
>
> > hi, all
> >
> > In my project, we want to consume the msg in kafka in the transaction,
> so,
> > when the transaction rollback, we need the kafka offset back to the
> > point transaction begin, I use the consumer Low-level API make it, it's
> too
> > complex, how to make Transaction get with the consumer High-level API if
> > is possible.
> >
> > Best regards,
> >
> > litao
> >
>
Re: [kafka-users] how to make Transaction get with the consumer
High-level API
Posted by Jun Rao <ju...@gmail.com>.
Tao,
You can use manual offset commit in the high level consumer. Basically,
call offset commit only after the data has been successfully processed.
Thanks,
Jun
On Mon, Jun 18, 2012 at 10:41 PM, tao li <tu...@gmail.com> wrote:
> hi, all
>
> In my project, we want to consume the msg in kafka in the transaction, so,
> when the transaction rollback, we need the kafka offset back to the
> point transaction begin, I use the consumer Low-level API make it, it's too
> complex, how to make Transaction get with the consumer High-level API if
> is possible.
>
> Best regards,
>
> litao
>