committing offsets

[Joris Peeters <j....@wintoncapital.com>]

I'm trying to set up a kafka consumer (in Java) that uses the new approach of committing offsets (i.e. in the __consumer_offsets topic etc, rather than through zookeeper).

Am I correct in believing that the current version (we're using kafka 8.2.1) does not expose this through the high level consumer yet?

I am using kafka.consumer.Consumer.createJavaConsumerConnector to set up the consumer. I can play with the enabling/disabling of auto-commits, or use commitOffsets to commit manually, but that does not seem to lead to any entries in __consumer_offsets. Also, by following the code trail a bit, it does seem as if offset committing is still happening through Zookeeper.

(One reason we'd like to use the new system is because we're interested in monitoring it through Burrow)

Apologies if I missed something simple.
Cheers!
-Joris.



Winton Capital Management Limited ("Winton") is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States.

This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email.

This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results.

Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.

RE: committing offsets

[Joris Peeters <j....@wintoncapital.com>]

Yep, that was it ...

Everything works now. And the only thing that didn't, earlier, was my head.
Thanks all!

-Joris.

-----Original Message-----
From: noah [mailto:iamnoah@gmail.com]
Sent: 22 September 2015 12:17
To: users@kafka.apache.org
Subject: Re: committing offsets

If you are using the console consumer to check the offsets topic, remember that you need this line in consumer.properties:
exclude.internal.topics=false

On Tue, Sep 22, 2015 at 6:05 AM Joris Peeters <j....@wintoncapital.com>
wrote:

> Ah, nice! Does not look like it is working, though. For some reason
> the __consumer_offsets topic is still empty. I see there's a few
> debug(..) logging messages that might get displayed if things go wrong
> - would you know how to get those displayed? (Right now I'm just
> running as 'java -jar ...' with no logging config).
>
> -J
>
> -----Original Message-----
> From: tao xiao [mailto:xiaotao183@gmail.com]
> Sent: 22 September 2015 10:51
> To: users@kafka.apache.org
> Subject: Re: committing offsets
>
> 0.8.2.1 already supports Kafka offset storage. You can set
> offsets.storage=kafka in consumer properties and high level API is
> able to pick it up and commit offsets to Kafka
>
> Here is the code reference where kafka offset logic kicks in
>
> https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/c
> onsumer/ZookeeperConsumerConnector.scala#L308
>
> On Tue, 22 Sep 2015 at 17:44 Joris Peeters
> <j....@wintoncapital.com>
> wrote:
>
> > I'm trying to set up a kafka consumer (in Java) that uses the new
> > approach of committing offsets (i.e. in the __consumer_offsets topic
> > etc, rather than through zookeeper).
> >
> > Am I correct in believing that the current version (we're using
> > kafka
> > 8.2.1) does not expose this through the high level consumer yet?
> >
> > I am using kafka.consumer.Consumer.createJavaConsumerConnector to
> > set up the consumer. I can play with the enabling/disabling of
> > auto-commits, or use commitOffsets to commit manually, but that does
> > not seem to lead to any entries in __consumer_offsets. Also, by
> > following the code trail a bit, it does seem as if offset committing
> > is
> still happening through Zookeeper.
> >
> > (One reason we'd like to use the new system is because we're
> > interested in monitoring it through Burrow)
> >
> > Apologies if I missed something simple.
> > Cheers!
> > -Joris.
> >
> >
> >
> > Winton Capital Management Limited ("Winton") is a limited company
> > registered in England and Wales with its registered offices at 16
> > Old Bailey, London, EC4M 7EG (Registered Company No. 3311531).
> > Winton is authorised and regulated by the Financial Conduct
> > Authority in the United Kingdom, registered as an investment adviser
> > with the US Securities and Exchange Commission, registered with the
> > US Commodity Futures Trading Commission and a member of the National
> > Futures Association in the United States.
> >
> > This communication, including any attachments, is confidential and
> > may be privileged. This email is for use by the intended recipient only.
> > If you receive it in error, please notify the sender and delete it.
> > You should not copy or disclose all or any part of this email.
> >
> > This email does not constitute an offer or solicitation and nothing
> > contained in this email constitutes, and should not be construed as,
> > investment advice. Prospective investors should request offering
> > materials and consult their own advisers with respect to investment
> > decisions and inform themselves as to applicable legal requirements,
> > exchange control regulations and taxes in the countries of their
> > citizenship, residence or domicile. Past performance is not
> > indicative
> of future results.
> >
> > Winton takes reasonable steps to ensure the accuracy and integrity
> > of its communications, including emails. However Winton accepts no
> > liability for any materials transmitted. Emails are not secure and
> > cannot be guaranteed to be error free.
> >
>
>
> Winton Capital Management Limited (“Winton”) is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the
> United Kingdom, registered as an investment adviser with the US
> Securities and Exchange Commission, registered with the US Commodity
> Futures Trading Commission and a member of the National Futures
> Association in the United States.
>
> This communication, including any attachments, is confidential and may
> be privileged. This email is for use by the intended recipient only.
> If you receive it in error, please notify the sender and delete it.
> You should not copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering
> materials and consult their own advisers with respect to investment
> decisions and inform themselves as to applicable legal requirements,
> exchange control regulations and taxes in the countries of their
> citizenship, residence or domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of
> its communications, including emails. However Winton accepts no
> liability for any materials transmitted. Emails are not secure and
> cannot be guaranteed to be error free.
>


Winton Capital Management Limited (“Winton”) is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States.

This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email.

This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results.

Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.

Re: committing offsets

[noah <ia...@gmail.com>]

If you are using the console consumer to check the offsets topic, remember
that you need this line in consumer.properties:
exclude.internal.topics=false

On Tue, Sep 22, 2015 at 6:05 AM Joris Peeters <j....@wintoncapital.com>
wrote:

> Ah, nice! Does not look like it is working, though. For some reason the
> __consumer_offsets topic is still empty. I see there's a few debug(..)
> logging messages that might get displayed if things go wrong - would you
> know how to get those displayed? (Right now I'm just running as 'java -jar
> ...' with no logging config).
>
> -J
>
> -----Original Message-----
> From: tao xiao [mailto:xiaotao183@gmail.com]
> Sent: 22 September 2015 10:51
> To: users@kafka.apache.org
> Subject: Re: committing offsets
>
> 0.8.2.1 already supports Kafka offset storage. You can set
> offsets.storage=kafka in consumer properties and high level API is able to
> pick it up and commit offsets to Kafka
>
> Here is the code reference where kafka offset logic kicks in
>
> https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L308
>
> On Tue, 22 Sep 2015 at 17:44 Joris Peeters <j....@wintoncapital.com>
> wrote:
>
> > I'm trying to set up a kafka consumer (in Java) that uses the new
> > approach of committing offsets (i.e. in the __consumer_offsets topic
> > etc, rather than through zookeeper).
> >
> > Am I correct in believing that the current version (we're using kafka
> > 8.2.1) does not expose this through the high level consumer yet?
> >
> > I am using kafka.consumer.Consumer.createJavaConsumerConnector to set
> > up the consumer. I can play with the enabling/disabling of
> > auto-commits, or use commitOffsets to commit manually, but that does
> > not seem to lead to any entries in __consumer_offsets. Also, by
> > following the code trail a bit, it does seem as if offset committing is
> still happening through Zookeeper.
> >
> > (One reason we'd like to use the new system is because we're
> > interested in monitoring it through Burrow)
> >
> > Apologies if I missed something simple.
> > Cheers!
> > -Joris.
> >
> >
> >
> > Winton Capital Management Limited ("Winton") is a limited company
> > registered in England and Wales with its registered offices at 16 Old
> > Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> > authorised and regulated by the Financial Conduct Authority in the
> > United Kingdom, registered as an investment adviser with the US
> > Securities and Exchange Commission, registered with the US Commodity
> > Futures Trading Commission and a member of the National Futures
> > Association in the United States.
> >
> > This communication, including any attachments, is confidential and may
> > be privileged. This email is for use by the intended recipient only.
> > If you receive it in error, please notify the sender and delete it.
> > You should not copy or disclose all or any part of this email.
> >
> > This email does not constitute an offer or solicitation and nothing
> > contained in this email constitutes, and should not be construed as,
> > investment advice. Prospective investors should request offering
> > materials and consult their own advisers with respect to investment
> > decisions and inform themselves as to applicable legal requirements,
> > exchange control regulations and taxes in the countries of their
> > citizenship, residence or domicile. Past performance is not indicative
> of future results.
> >
> > Winton takes reasonable steps to ensure the accuracy and integrity of
> > its communications, including emails. However Winton accepts no
> > liability for any materials transmitted. Emails are not secure and
> > cannot be guaranteed to be error free.
> >
>
>
> Winton Capital Management Limited (“Winton”) is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the United
> Kingdom, registered as an investment adviser with the US Securities and
> Exchange Commission, registered with the US Commodity Futures Trading
> Commission and a member of the National Futures Association in the United
> States.
>
> This communication, including any attachments, is confidential and may be
> privileged. This email is for use by the intended recipient only. If you
> receive it in error, please notify the sender and delete it. You should not
> copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering materials
> and consult their own advisers with respect to investment decisions and
> inform themselves as to applicable legal requirements, exchange control
> regulations and taxes in the countries of their citizenship, residence or
> domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of its
> communications, including emails. However Winton accepts no liability for
> any materials transmitted. Emails are not secure and cannot be guaranteed
> to be error free.
>

RE: committing offsets

[Joris Peeters <j....@wintoncapital.com>]

Ah, nice! Does not look like it is working, though. For some reason the __consumer_offsets topic is still empty. I see there's a few debug(..) logging messages that might get displayed if things go wrong - would you know how to get those displayed? (Right now I'm just running as 'java -jar ...' with no logging config).

-J

-----Original Message-----
From: tao xiao [mailto:xiaotao183@gmail.com]
Sent: 22 September 2015 10:51
To: users@kafka.apache.org
Subject: Re: committing offsets

0.8.2.1 already supports Kafka offset storage. You can set offsets.storage=kafka in consumer properties and high level API is able to pick it up and commit offsets to Kafka

Here is the code reference where kafka offset logic kicks in
https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L308

On Tue, 22 Sep 2015 at 17:44 Joris Peeters <j....@wintoncapital.com>
wrote:

> I'm trying to set up a kafka consumer (in Java) that uses the new
> approach of committing offsets (i.e. in the __consumer_offsets topic
> etc, rather than through zookeeper).
>
> Am I correct in believing that the current version (we're using kafka
> 8.2.1) does not expose this through the high level consumer yet?
>
> I am using kafka.consumer.Consumer.createJavaConsumerConnector to set
> up the consumer. I can play with the enabling/disabling of
> auto-commits, or use commitOffsets to commit manually, but that does
> not seem to lead to any entries in __consumer_offsets. Also, by
> following the code trail a bit, it does seem as if offset committing is still happening through Zookeeper.
>
> (One reason we'd like to use the new system is because we're
> interested in monitoring it through Burrow)
>
> Apologies if I missed something simple.
> Cheers!
> -Joris.
>
>
>
> Winton Capital Management Limited ("Winton") is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the
> United Kingdom, registered as an investment adviser with the US
> Securities and Exchange Commission, registered with the US Commodity
> Futures Trading Commission and a member of the National Futures
> Association in the United States.
>
> This communication, including any attachments, is confidential and may
> be privileged. This email is for use by the intended recipient only.
> If you receive it in error, please notify the sender and delete it.
> You should not copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering
> materials and consult their own advisers with respect to investment
> decisions and inform themselves as to applicable legal requirements,
> exchange control regulations and taxes in the countries of their
> citizenship, residence or domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of
> its communications, including emails. However Winton accepts no
> liability for any materials transmitted. Emails are not secure and
> cannot be guaranteed to be error free.
>


Winton Capital Management Limited (“Winton”) is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States.

This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email.

This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results.

Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.

Re: committing offsets

[tao xiao <xi...@gmail.com>]

0.8.2.1 already supports Kafka offset storage. You can
set offsets.storage=kafka in consumer properties and high level API is able
to pick it up and commit offsets to Kafka

Here is the code reference where kafka offset logic kicks in
https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L308

On Tue, 22 Sep 2015 at 17:44 Joris Peeters <j....@wintoncapital.com>
wrote:

> I'm trying to set up a kafka consumer (in Java) that uses the new approach
> of committing offsets (i.e. in the __consumer_offsets topic etc, rather
> than through zookeeper).
>
> Am I correct in believing that the current version (we're using kafka
> 8.2.1) does not expose this through the high level consumer yet?
>
> I am using kafka.consumer.Consumer.createJavaConsumerConnector to set up
> the consumer. I can play with the enabling/disabling of auto-commits, or
> use commitOffsets to commit manually, but that does not seem to lead to any
> entries in __consumer_offsets. Also, by following the code trail a bit, it
> does seem as if offset committing is still happening through Zookeeper.
>
> (One reason we'd like to use the new system is because we're interested in
> monitoring it through Burrow)
>
> Apologies if I missed something simple.
> Cheers!
> -Joris.
>
>
>
> Winton Capital Management Limited ("Winton") is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the United
> Kingdom, registered as an investment adviser with the US Securities and
> Exchange Commission, registered with the US Commodity Futures Trading
> Commission and a member of the National Futures Association in the United
> States.
>
> This communication, including any attachments, is confidential and may be
> privileged. This email is for use by the intended recipient only. If you
> receive it in error, please notify the sender and delete it. You should not
> copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering materials
> and consult their own advisers with respect to investment decisions and
> inform themselves as to applicable legal requirements, exchange control
> regulations and taxes in the countries of their citizenship, residence or
> domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of its
> communications, including emails. However Winton accepts no liability for
> any materials transmitted. Emails are not secure and cannot be guaranteed
> to be error free.
>