You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dana Powers <da...@rd.io> on 2014/09/01 02:20:32 UTC
Re: offset commit api
following up on this -- I think the online API docs for OffsetCommitRequest
still incorrectly refer to client-side timestamps:
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommitRequest
Wasn't that removed and now always handled server-side now? Would one of
the devs mind updating the API spec wiki?
-Dana
On Mon, Aug 18, 2014 at 10:41 AM, Joel Koshy <jj...@gmail.com> wrote:
> If you are using the high-level consumer then yes you would be under a
> group. If you are using the simple consumer then no. I just saw your
> earlier email in this thread - if you are using the simple consumer
> you can still use the offset commit API and set a group manually.
>
> On Mon, Aug 18, 2014 at 09:24:06AM -0700, Weide Zhang wrote:
> > Thanks Joel. Do you know if I'm using consumer group string
> >
> >
> > On Mon, Aug 4, 2014 at 3:16 PM, Joel Koshy <jj...@gmail.com> wrote:
> > > Weide, 0.8.1.1 does not support offsets storage in Kafka. The brokers
> > > do support offset commit requests/fetches but simply forward to
> > > ZooKeeper - you can issue the offset commit and fetch requests to any
> > > broker. Kafka-backed consumer offsets is currently in trunk and will
> > > be released in 0.8.2.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Mon, Aug 04, 2014 at 02:57:02PM -0700, Weide Zhang wrote:
> > > > Hi
> > > >
> > > > It seems to me that 0.8.1.1 doesn't have the ConsumerMetadata API.
> So what
> > > > broker I should choose in order to commit and fetch offset
> information ?
> > > > Shall I use zookeeper for offset to manage it manually instead ?
> > > >
> > > > Thanks,
> > > >
> > > > Weide
> > > >
> > > >
> > > >
> > > > On Sun, Aug 3, 2014 at 4:34 PM, Weide Zhang <we...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm reading the offset management on the API link.
> > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommit/FetchAPI
> > > > >
> > > > > I have a couple of questions regarding using the offset fetch and
> commit
> > > > > API in 0.8.1.1 ?
> > > > >
> > > > > 1. Is the new offset commit and fetch api usable in use in 0.8.1.1
> ? Does
> > > > > 0.8.1.1 already support offset coordinator ?
> > > > >
> > > > > 2. what's the difference between old offsetrequest and new
> > > > > offsetfetchrequest ? It seems to me that the new api support per
> consumer
> > > > > group offset management fetch while old api doesn't. Also, what's
> the
> > > > > purpose of using a timestamp parameter in the fetch request ?
> > > > >
> > > > > 3. In 0.8.1.1, the OffsetCommitRequest uses
> OffsetMetadataAndError, could
> > > > > you tell me what's the purpose of the error parameter and metadata
> > > > > parameter in the request ?
> > > > >
> > > > > 4. Can I assume the offset management is purely independent of
> message
> > > > > consumption ? In other words, if I use a simple consumer to fetch
> message
> > > > > with random client id, can I still manually set some consumer
> group along
> > > > > with offset in the offset commit message ? Is that allowed ?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Weide
> > > > >
> >
>
>
Re: offset commit api
Posted by Neha Narkhede <ne...@gmail.com>.
Filed https://issues.apache.org/jira/browse/KAFKA-1634 to track the change
and marked it for 0.8.2.
On Sun, Aug 31, 2014 at 5:20 PM, Dana Powers <da...@rd.io> wrote:
> following up on this -- I think the online API docs for OffsetCommitRequest
> still incorrectly refer to client-side timestamps:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommitRequest
>
> Wasn't that removed and now always handled server-side now? Would one of
> the devs mind updating the API spec wiki?
>
> -Dana
>
>
> On Mon, Aug 18, 2014 at 10:41 AM, Joel Koshy <jj...@gmail.com> wrote:
>
> > If you are using the high-level consumer then yes you would be under a
> > group. If you are using the simple consumer then no. I just saw your
> > earlier email in this thread - if you are using the simple consumer
> > you can still use the offset commit API and set a group manually.
> >
> > On Mon, Aug 18, 2014 at 09:24:06AM -0700, Weide Zhang wrote:
> > > Thanks Joel. Do you know if I'm using consumer group string
> > >
> > >
> > > On Mon, Aug 4, 2014 at 3:16 PM, Joel Koshy <jj...@gmail.com>
> wrote:
> > > > Weide, 0.8.1.1 does not support offsets storage in Kafka. The brokers
> > > > do support offset commit requests/fetches but simply forward to
> > > > ZooKeeper - you can issue the offset commit and fetch requests to any
> > > > broker. Kafka-backed consumer offsets is currently in trunk and will
> > > > be released in 0.8.2.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > > > On Mon, Aug 04, 2014 at 02:57:02PM -0700, Weide Zhang wrote:
> > > > > Hi
> > > > >
> > > > > It seems to me that 0.8.1.1 doesn't have the ConsumerMetadata API.
> > So what
> > > > > broker I should choose in order to commit and fetch offset
> > information ?
> > > > > Shall I use zookeeper for offset to manage it manually instead ?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Weide
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Aug 3, 2014 at 4:34 PM, Weide Zhang <we...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm reading the offset management on the API link.
> > > > > >
> > > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommit/FetchAPI
> > > > > >
> > > > > > I have a couple of questions regarding using the offset fetch and
> > commit
> > > > > > API in 0.8.1.1 ?
> > > > > >
> > > > > > 1. Is the new offset commit and fetch api usable in use in
> 0.8.1.1
> > ? Does
> > > > > > 0.8.1.1 already support offset coordinator ?
> > > > > >
> > > > > > 2. what's the difference between old offsetrequest and new
> > > > > > offsetfetchrequest ? It seems to me that the new api support per
> > consumer
> > > > > > group offset management fetch while old api doesn't. Also, what's
> > the
> > > > > > purpose of using a timestamp parameter in the fetch request ?
> > > > > >
> > > > > > 3. In 0.8.1.1, the OffsetCommitRequest uses
> > OffsetMetadataAndError, could
> > > > > > you tell me what's the purpose of the error parameter and
> metadata
> > > > > > parameter in the request ?
> > > > > >
> > > > > > 4. Can I assume the offset management is purely independent of
> > message
> > > > > > consumption ? In other words, if I use a simple consumer to fetch
> > message
> > > > > > with random client id, can I still manually set some consumer
> > group along
> > > > > > with offset in the offset commit message ? Is that allowed ?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Weide
> > > > > >
> > >
> >
> >
>