You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Aaron Niskode-Dossett <an...@etsy.com.INVALID> on 2017/03/15 00:47:32 UTC
Kafka connection to start from latest offset
Is it possible to start a kafka connect instance that reads from the
*latest* offset as opposed to the earliest? I suppose this would be the
equivalent of passing auto.offset.reset=earliest to a kafka consumer.
More generally, is this something that specific implementations of the
kafka connect API would have to be responsible for handling or should it
exposed through the connect API?
Thanks, Aaron
Re: Kafka connection to start from latest offset
Posted by Stephen Durfey <sj...@gmail.com>.
Yes, it is pretty coarse. I have a pull request open for supporting
overriding those settings at the connector level, I'm just waiting for it
to be pulled.
So, if any committers are interested in reviewing/pulling it for me, that
would be great :)
https://github.com/apache/kafka/pull/2548
On Wed, Mar 15, 2017 at 8:42 AM, Aaron Niskode-Dossett <
aniskodedossett@etsy.com.invalid> wrote:
> Thank you Stephen! That's a very coarse setting, as you note, since it's
> at the worker level, but I'll take it.
>
> -Aaron
>
> On Tue, Mar 14, 2017 at 8:07 PM Stephen Durfey <sj...@gmail.com> wrote:
>
> > Producer and consumer overrides used by the connect worker can be
> > overridden by prefixing the specific kafka config with either 'producer.'
> > or 'consumer.'. So, you should be able to set
> > 'consumer.auto.offset.reset=latest' in your worker config to do that.
> >
> >
> > http://docs.confluent.io/3.0.0/connect/userguide.html?
> highlight=override%20configuration#overriding-producer-consumer-settings
> >
> > On Tue, Mar 14, 2017 at 7:47 PM, Aaron Niskode-Dossett <
> > aniskodedossett@etsy.com.invalid> wrote:
> >
> > > Is it possible to start a kafka connect instance that reads from the
> > > *latest* offset as opposed to the earliest? I suppose this would be
> the
> > > equivalent of passing auto.offset.reset=earliest to a kafka consumer.
> > >
> > > More generally, is this something that specific implementations of the
> > > kafka connect API would have to be responsible for handling or should
> it
> > > exposed through the connect API?
> > >
> > > Thanks, Aaron
> > >
> >
>
Re: Kafka connection to start from latest offset
Posted by Aaron Niskode-Dossett <an...@etsy.com.INVALID>.
Thank you Stephen! That's a very coarse setting, as you note, since it's
at the worker level, but I'll take it.
-Aaron
On Tue, Mar 14, 2017 at 8:07 PM Stephen Durfey <sj...@gmail.com> wrote:
> Producer and consumer overrides used by the connect worker can be
> overridden by prefixing the specific kafka config with either 'producer.'
> or 'consumer.'. So, you should be able to set
> 'consumer.auto.offset.reset=latest' in your worker config to do that.
>
>
> http://docs.confluent.io/3.0.0/connect/userguide.html?highlight=override%20configuration#overriding-producer-consumer-settings
>
> On Tue, Mar 14, 2017 at 7:47 PM, Aaron Niskode-Dossett <
> aniskodedossett@etsy.com.invalid> wrote:
>
> > Is it possible to start a kafka connect instance that reads from the
> > *latest* offset as opposed to the earliest? I suppose this would be the
> > equivalent of passing auto.offset.reset=earliest to a kafka consumer.
> >
> > More generally, is this something that specific implementations of the
> > kafka connect API would have to be responsible for handling or should it
> > exposed through the connect API?
> >
> > Thanks, Aaron
> >
>
Re: Kafka connection to start from latest offset
Posted by Stephen Durfey <sj...@gmail.com>.
Producer and consumer overrides used by the connect worker can be
overridden by prefixing the specific kafka config with either 'producer.'
or 'consumer.'. So, you should be able to set
'consumer.auto.offset.reset=latest' in your worker config to do that.
http://docs.confluent.io/3.0.0/connect/userguide.html?highlight=override%20configuration#overriding-producer-consumer-settings
On Tue, Mar 14, 2017 at 7:47 PM, Aaron Niskode-Dossett <
aniskodedossett@etsy.com.invalid> wrote:
> Is it possible to start a kafka connect instance that reads from the
> *latest* offset as opposed to the earliest? I suppose this would be the
> equivalent of passing auto.offset.reset=earliest to a kafka consumer.
>
> More generally, is this something that specific implementations of the
> kafka connect API would have to be responsible for handling or should it
> exposed through the connect API?
>
> Thanks, Aaron
>