You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sameer Kumar <sa...@gmail.com> on 2017/10/03 06:29:16 UTC
Re: out of order sequence number in exactly once streams
I had enabled eos through streams config.
On Fri, Sep 29, 2017 at 11:12 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> That's correct: If EOS is enabled, we enforce some producer configs:
>
> https://github.com/apache/kafka/blob/0.11.0.1/streams/
> src/main/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688
>
> https://github.com/apache/kafka/blob/0.11.0.1/streams/
> src/main/java/org/apache/kafka/streams/StreamsConfig.java#L691
>
> https://github.com/apache/kafka/blob/0.11.0.1/streams/
> src/main/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496
>
>
> Note, that by default we set retries to Integer.MAX_VALUE but we do not
> enforce this setting (as pointed out by Damian already). So you could
> overwrite it with a smaller value (what is of course not recommended).
>
> I was not sure though, if you enabled EOS or just enabled idempotency
> only for the producer -- what you can easily do by providing the
> corresponding producer configs.
>
> If you did enable EOS in StreamsConfig, Producer will take care of
> OutOfOrderSequenceException in general.
>
> However, there are scenarios for which Producer cannot handle it. For
> those case, OutOfOrderSequenceException indicates that there has been
> data loss on the broker, ie, a previously acknowledged message no longer
> exists. For most part, this should only occur in rare situations
> (simultaneous power outages, multiple disk losses, software bugs
> resulting in data corruption, etc.).
>
>
> -Matthias
>
> On 9/29/17 7:55 AM, Damian Guy wrote:
> > You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
> >
> > Properties props = new Properties();
> > props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
> > ...
> >
> > On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com>
> wrote:
> >
> >> I guess once stream app are enabled exactly-once, producer idempotence
> get
> >> enabled by default and so do the retries. I guess producer retries are
> >> managed internally and not exposed through streamconfig.
> >>
> >> https://kafka.apache.org/0110/documentation/#streamsconfigs
> >>
> >> -Sameer.
> >>
> >> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <
> matthias@confluent.io>
> >> wrote:
> >>
> >>> An OutOfOrderSequenceException should only occur if a idempotent
> >>> producer gets out of sync with the broker. If you set
> >>> `enable.idempotence = true` on your producer, you might want to set
> >>> `retries = Integer.MAX_VALUE`.
> >>>
> >>> -Matthias
> >>>
> >>> On 9/26/17 11:30 PM, Sameer Kumar wrote:
> >>>> Hi,
> >>>>
> >>>> I again received this exception while running my streams app. I am
> >> using
> >>>> Kafka 11.0.1. After restarting my app, this error got fixed.
> >>>>
> >>>> I guess this might be due to bad network. Any pointers. Any config
> >>>> wherein I can configure it for retries.
> >>>>
> >>>> Exception trace is attached.
> >>>>
> >>>> Regards,
> >>>> -Sameer.
> >>>
> >>>
> >>
> >
>
>
Re: out of order sequence number in exactly once streams
Posted by Ismael Juma <is...@juma.me.uk>.
I suggest filing a JIRA with as much information as possible (logs,
configs, code, etc.) so that we can try to reproduce it.
Ismael
On Wed, Oct 4, 2017 at 9:48 AM, Sameer Kumar <sa...@gmail.com> wrote:
> I am receiving this exception lot more frequently, also this occurs
> generally at the start of starting the streams app. I was also running 3
> seperate instances of my streams app all with different state stores on 3
> different machines.
>
> I am using Kafka 11.0.1.
>
> -Sameer.
>
> On Tue, Oct 3, 2017 at 12:03 PM, Sameer Kumar <sa...@gmail.com>
> wrote:
>
> > I had enabled eos through streams config and as explained in the
> > documentation, I have not added anything else other than following
> config.
> >
> > streamsConfiguration.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> > StreamsConfig.EXACTLY_ONCE);
> >
> > As explained by you, I think producer idempotence and retries would be
> > automatically picked up. I was thinking if there is someway i can print
> the
> > configs picked up the streams app for e.g. the current retries.
> >
> > Do i need to add any other configuration.
> >
> > -Sameer.
> >
> > On Tue, Oct 3, 2017 at 11:59 AM, Sameer Kumar <sa...@gmail.com>
> > wrote:
> >
> >> I had enabled eos through streams config.
> >>
> >>
> >> On Fri, Sep 29, 2017 at 11:12 PM, Matthias J. Sax <
> matthias@confluent.io>
> >> wrote:
> >>
> >>> That's correct: If EOS is enabled, we enforce some producer configs:
> >>>
> >>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
> >>> in/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688
> >>>
> >>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
> >>> in/java/org/apache/kafka/streams/StreamsConfig.java#L691
> >>>
> >>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
> >>> in/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496
> >>>
> >>>
> >>> Note, that by default we set retries to Integer.MAX_VALUE but we do not
> >>> enforce this setting (as pointed out by Damian already). So you could
> >>> overwrite it with a smaller value (what is of course not recommended).
> >>>
> >>> I was not sure though, if you enabled EOS or just enabled idempotency
> >>> only for the producer -- what you can easily do by providing the
> >>> corresponding producer configs.
> >>>
> >>> If you did enable EOS in StreamsConfig, Producer will take care of
> >>> OutOfOrderSequenceException in general.
> >>>
> >>> However, there are scenarios for which Producer cannot handle it. For
> >>> those case, OutOfOrderSequenceException indicates that there has been
> >>> data loss on the broker, ie, a previously acknowledged message no
> longer
> >>> exists. For most part, this should only occur in rare situations
> >>> (simultaneous power outages, multiple disk losses, software bugs
> >>> resulting in data corruption, etc.).
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 9/29/17 7:55 AM, Damian Guy wrote:
> >>> > You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
> >>> >
> >>> > Properties props = new Properties();
> >>> > props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
> >>> > ...
> >>> >
> >>> > On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com>
> >>> wrote:
> >>> >
> >>> >> I guess once stream app are enabled exactly-once, producer
> >>> idempotence get
> >>> >> enabled by default and so do the retries. I guess producer retries
> are
> >>> >> managed internally and not exposed through streamconfig.
> >>> >>
> >>> >> https://kafka.apache.org/0110/documentation/#streamsconfigs
> >>> >>
> >>> >> -Sameer.
> >>> >>
> >>> >> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <
> >>> matthias@confluent.io>
> >>> >> wrote:
> >>> >>
> >>> >>> An OutOfOrderSequenceException should only occur if a idempotent
> >>> >>> producer gets out of sync with the broker. If you set
> >>> >>> `enable.idempotence = true` on your producer, you might want to set
> >>> >>> `retries = Integer.MAX_VALUE`.
> >>> >>>
> >>> >>> -Matthias
> >>> >>>
> >>> >>> On 9/26/17 11:30 PM, Sameer Kumar wrote:
> >>> >>>> Hi,
> >>> >>>>
> >>> >>>> I again received this exception while running my streams app. I am
> >>> >> using
> >>> >>>> Kafka 11.0.1. After restarting my app, this error got fixed.
> >>> >>>>
> >>> >>>> I guess this might be due to bad network. Any pointers. Any config
> >>> >>>> wherein I can configure it for retries.
> >>> >>>>
> >>> >>>> Exception trace is attached.
> >>> >>>>
> >>> >>>> Regards,
> >>> >>>> -Sameer.
> >>> >>>
> >>> >>>
> >>> >>
> >>> >
> >>>
> >>>
> >>
> >
>
Re: out of order sequence number in exactly once streams
Posted by Sameer Kumar <sa...@gmail.com>.
I am receiving this exception lot more frequently, also this occurs
generally at the start of starting the streams app. I was also running 3
seperate instances of my streams app all with different state stores on 3
different machines.
I am using Kafka 11.0.1.
-Sameer.
On Tue, Oct 3, 2017 at 12:03 PM, Sameer Kumar <sa...@gmail.com>
wrote:
> I had enabled eos through streams config and as explained in the
> documentation, I have not added anything else other than following config.
>
> streamsConfiguration.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.EXACTLY_ONCE);
>
> As explained by you, I think producer idempotence and retries would be
> automatically picked up. I was thinking if there is someway i can print the
> configs picked up the streams app for e.g. the current retries.
>
> Do i need to add any other configuration.
>
> -Sameer.
>
> On Tue, Oct 3, 2017 at 11:59 AM, Sameer Kumar <sa...@gmail.com>
> wrote:
>
>> I had enabled eos through streams config.
>>
>>
>> On Fri, Sep 29, 2017 at 11:12 PM, Matthias J. Sax <ma...@confluent.io>
>> wrote:
>>
>>> That's correct: If EOS is enabled, we enforce some producer configs:
>>>
>>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
>>> in/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688
>>>
>>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
>>> in/java/org/apache/kafka/streams/StreamsConfig.java#L691
>>>
>>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
>>> in/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496
>>>
>>>
>>> Note, that by default we set retries to Integer.MAX_VALUE but we do not
>>> enforce this setting (as pointed out by Damian already). So you could
>>> overwrite it with a smaller value (what is of course not recommended).
>>>
>>> I was not sure though, if you enabled EOS or just enabled idempotency
>>> only for the producer -- what you can easily do by providing the
>>> corresponding producer configs.
>>>
>>> If you did enable EOS in StreamsConfig, Producer will take care of
>>> OutOfOrderSequenceException in general.
>>>
>>> However, there are scenarios for which Producer cannot handle it. For
>>> those case, OutOfOrderSequenceException indicates that there has been
>>> data loss on the broker, ie, a previously acknowledged message no longer
>>> exists. For most part, this should only occur in rare situations
>>> (simultaneous power outages, multiple disk losses, software bugs
>>> resulting in data corruption, etc.).
>>>
>>>
>>> -Matthias
>>>
>>> On 9/29/17 7:55 AM, Damian Guy wrote:
>>> > You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
>>> >
>>> > Properties props = new Properties();
>>> > props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
>>> > ...
>>> >
>>> > On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com>
>>> wrote:
>>> >
>>> >> I guess once stream app are enabled exactly-once, producer
>>> idempotence get
>>> >> enabled by default and so do the retries. I guess producer retries are
>>> >> managed internally and not exposed through streamconfig.
>>> >>
>>> >> https://kafka.apache.org/0110/documentation/#streamsconfigs
>>> >>
>>> >> -Sameer.
>>> >>
>>> >> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <
>>> matthias@confluent.io>
>>> >> wrote:
>>> >>
>>> >>> An OutOfOrderSequenceException should only occur if a idempotent
>>> >>> producer gets out of sync with the broker. If you set
>>> >>> `enable.idempotence = true` on your producer, you might want to set
>>> >>> `retries = Integer.MAX_VALUE`.
>>> >>>
>>> >>> -Matthias
>>> >>>
>>> >>> On 9/26/17 11:30 PM, Sameer Kumar wrote:
>>> >>>> Hi,
>>> >>>>
>>> >>>> I again received this exception while running my streams app. I am
>>> >> using
>>> >>>> Kafka 11.0.1. After restarting my app, this error got fixed.
>>> >>>>
>>> >>>> I guess this might be due to bad network. Any pointers. Any config
>>> >>>> wherein I can configure it for retries.
>>> >>>>
>>> >>>> Exception trace is attached.
>>> >>>>
>>> >>>> Regards,
>>> >>>> -Sameer.
>>> >>>
>>> >>>
>>> >>
>>> >
>>>
>>>
>>
>
Re: out of order sequence number in exactly once streams
Posted by Sameer Kumar <sa...@gmail.com>.
I had enabled eos through streams config and as explained in the
documentation, I have not added anything else other than following config.
streamsConfiguration.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
StreamsConfig.EXACTLY_ONCE);
As explained by you, I think producer idempotence and retries would be
automatically picked up. I was thinking if there is someway i can print the
configs picked up the streams app for e.g. the current retries.
Do i need to add any other configuration.
-Sameer.
On Tue, Oct 3, 2017 at 11:59 AM, Sameer Kumar <sa...@gmail.com>
wrote:
> I had enabled eos through streams config.
>
>
> On Fri, Sep 29, 2017 at 11:12 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> That's correct: If EOS is enabled, we enforce some producer configs:
>>
>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/
>> main/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688
>>
>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/
>> main/java/org/apache/kafka/streams/StreamsConfig.java#L691
>>
>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/
>> main/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496
>>
>>
>> Note, that by default we set retries to Integer.MAX_VALUE but we do not
>> enforce this setting (as pointed out by Damian already). So you could
>> overwrite it with a smaller value (what is of course not recommended).
>>
>> I was not sure though, if you enabled EOS or just enabled idempotency
>> only for the producer -- what you can easily do by providing the
>> corresponding producer configs.
>>
>> If you did enable EOS in StreamsConfig, Producer will take care of
>> OutOfOrderSequenceException in general.
>>
>> However, there are scenarios for which Producer cannot handle it. For
>> those case, OutOfOrderSequenceException indicates that there has been
>> data loss on the broker, ie, a previously acknowledged message no longer
>> exists. For most part, this should only occur in rare situations
>> (simultaneous power outages, multiple disk losses, software bugs
>> resulting in data corruption, etc.).
>>
>>
>> -Matthias
>>
>> On 9/29/17 7:55 AM, Damian Guy wrote:
>> > You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
>> >
>> > Properties props = new Properties();
>> > props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
>> > ...
>> >
>> > On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com>
>> wrote:
>> >
>> >> I guess once stream app are enabled exactly-once, producer idempotence
>> get
>> >> enabled by default and so do the retries. I guess producer retries are
>> >> managed internally and not exposed through streamconfig.
>> >>
>> >> https://kafka.apache.org/0110/documentation/#streamsconfigs
>> >>
>> >> -Sameer.
>> >>
>> >> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <
>> matthias@confluent.io>
>> >> wrote:
>> >>
>> >>> An OutOfOrderSequenceException should only occur if a idempotent
>> >>> producer gets out of sync with the broker. If you set
>> >>> `enable.idempotence = true` on your producer, you might want to set
>> >>> `retries = Integer.MAX_VALUE`.
>> >>>
>> >>> -Matthias
>> >>>
>> >>> On 9/26/17 11:30 PM, Sameer Kumar wrote:
>> >>>> Hi,
>> >>>>
>> >>>> I again received this exception while running my streams app. I am
>> >> using
>> >>>> Kafka 11.0.1. After restarting my app, this error got fixed.
>> >>>>
>> >>>> I guess this might be due to bad network. Any pointers. Any config
>> >>>> wherein I can configure it for retries.
>> >>>>
>> >>>> Exception trace is attached.
>> >>>>
>> >>>> Regards,
>> >>>> -Sameer.
>> >>>
>> >>>
>> >>
>> >
>>
>>
>