You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sameer Kumar <sa...@gmail.com> on 2017/09/27 06:30:20 UTC
out of order sequence number in exactly once streams
Hi,
I again received this exception while running my streams app. I am using
Kafka 11.0.1. After restarting my app, this error got fixed.
I guess this might be due to bad network. Any pointers. Any config wherein
I can configure it for retries.
Exception trace is attached.
Regards,
-Sameer.
Re: out of order sequence number in exactly once streams
Posted by Ismael Juma <is...@juma.me.uk>.
I suggest filing a JIRA with as much information as possible (logs,
configs, code, etc.) so that we can try to reproduce it.
Ismael
On Wed, Oct 4, 2017 at 9:48 AM, Sameer Kumar <sa...@gmail.com> wrote:
> I am receiving this exception lot more frequently, also this occurs
> generally at the start of starting the streams app. I was also running 3
> seperate instances of my streams app all with different state stores on 3
> different machines.
>
> I am using Kafka 11.0.1.
>
> -Sameer.
>
> On Tue, Oct 3, 2017 at 12:03 PM, Sameer Kumar <sa...@gmail.com>
> wrote:
>
> > I had enabled eos through streams config and as explained in the
> > documentation, I have not added anything else other than following
> config.
> >
> > streamsConfiguration.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> > StreamsConfig.EXACTLY_ONCE);
> >
> > As explained by you, I think producer idempotence and retries would be
> > automatically picked up. I was thinking if there is someway i can print
> the
> > configs picked up the streams app for e.g. the current retries.
> >
> > Do i need to add any other configuration.
> >
> > -Sameer.
> >
> > On Tue, Oct 3, 2017 at 11:59 AM, Sameer Kumar <sa...@gmail.com>
> > wrote:
> >
> >> I had enabled eos through streams config.
> >>
> >>
> >> On Fri, Sep 29, 2017 at 11:12 PM, Matthias J. Sax <
> matthias@confluent.io>
> >> wrote:
> >>
> >>> That's correct: If EOS is enabled, we enforce some producer configs:
> >>>
> >>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
> >>> in/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688
> >>>
> >>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
> >>> in/java/org/apache/kafka/streams/StreamsConfig.java#L691
> >>>
> >>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
> >>> in/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496
> >>>
> >>>
> >>> Note, that by default we set retries to Integer.MAX_VALUE but we do not
> >>> enforce this setting (as pointed out by Damian already). So you could
> >>> overwrite it with a smaller value (what is of course not recommended).
> >>>
> >>> I was not sure though, if you enabled EOS or just enabled idempotency
> >>> only for the producer -- what you can easily do by providing the
> >>> corresponding producer configs.
> >>>
> >>> If you did enable EOS in StreamsConfig, Producer will take care of
> >>> OutOfOrderSequenceException in general.
> >>>
> >>> However, there are scenarios for which Producer cannot handle it. For
> >>> those case, OutOfOrderSequenceException indicates that there has been
> >>> data loss on the broker, ie, a previously acknowledged message no
> longer
> >>> exists. For most part, this should only occur in rare situations
> >>> (simultaneous power outages, multiple disk losses, software bugs
> >>> resulting in data corruption, etc.).
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 9/29/17 7:55 AM, Damian Guy wrote:
> >>> > You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
> >>> >
> >>> > Properties props = new Properties();
> >>> > props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
> >>> > ...
> >>> >
> >>> > On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com>
> >>> wrote:
> >>> >
> >>> >> I guess once stream app are enabled exactly-once, producer
> >>> idempotence get
> >>> >> enabled by default and so do the retries. I guess producer retries
> are
> >>> >> managed internally and not exposed through streamconfig.
> >>> >>
> >>> >> https://kafka.apache.org/0110/documentation/#streamsconfigs
> >>> >>
> >>> >> -Sameer.
> >>> >>
> >>> >> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <
> >>> matthias@confluent.io>
> >>> >> wrote:
> >>> >>
> >>> >>> An OutOfOrderSequenceException should only occur if a idempotent
> >>> >>> producer gets out of sync with the broker. If you set
> >>> >>> `enable.idempotence = true` on your producer, you might want to set
> >>> >>> `retries = Integer.MAX_VALUE`.
> >>> >>>
> >>> >>> -Matthias
> >>> >>>
> >>> >>> On 9/26/17 11:30 PM, Sameer Kumar wrote:
> >>> >>>> Hi,
> >>> >>>>
> >>> >>>> I again received this exception while running my streams app. I am
> >>> >> using
> >>> >>>> Kafka 11.0.1. After restarting my app, this error got fixed.
> >>> >>>>
> >>> >>>> I guess this might be due to bad network. Any pointers. Any config
> >>> >>>> wherein I can configure it for retries.
> >>> >>>>
> >>> >>>> Exception trace is attached.
> >>> >>>>
> >>> >>>> Regards,
> >>> >>>> -Sameer.
> >>> >>>
> >>> >>>
> >>> >>
> >>> >
> >>>
> >>>
> >>
> >
>
Re: out of order sequence number in exactly once streams
Posted by Sameer Kumar <sa...@gmail.com>.
I am receiving this exception lot more frequently, also this occurs
generally at the start of starting the streams app. I was also running 3
seperate instances of my streams app all with different state stores on 3
different machines.
I am using Kafka 11.0.1.
-Sameer.
On Tue, Oct 3, 2017 at 12:03 PM, Sameer Kumar <sa...@gmail.com>
wrote:
> I had enabled eos through streams config and as explained in the
> documentation, I have not added anything else other than following config.
>
> streamsConfiguration.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.EXACTLY_ONCE);
>
> As explained by you, I think producer idempotence and retries would be
> automatically picked up. I was thinking if there is someway i can print the
> configs picked up the streams app for e.g. the current retries.
>
> Do i need to add any other configuration.
>
> -Sameer.
>
> On Tue, Oct 3, 2017 at 11:59 AM, Sameer Kumar <sa...@gmail.com>
> wrote:
>
>> I had enabled eos through streams config.
>>
>>
>> On Fri, Sep 29, 2017 at 11:12 PM, Matthias J. Sax <ma...@confluent.io>
>> wrote:
>>
>>> That's correct: If EOS is enabled, we enforce some producer configs:
>>>
>>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
>>> in/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688
>>>
>>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
>>> in/java/org/apache/kafka/streams/StreamsConfig.java#L691
>>>
>>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ma
>>> in/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496
>>>
>>>
>>> Note, that by default we set retries to Integer.MAX_VALUE but we do not
>>> enforce this setting (as pointed out by Damian already). So you could
>>> overwrite it with a smaller value (what is of course not recommended).
>>>
>>> I was not sure though, if you enabled EOS or just enabled idempotency
>>> only for the producer -- what you can easily do by providing the
>>> corresponding producer configs.
>>>
>>> If you did enable EOS in StreamsConfig, Producer will take care of
>>> OutOfOrderSequenceException in general.
>>>
>>> However, there are scenarios for which Producer cannot handle it. For
>>> those case, OutOfOrderSequenceException indicates that there has been
>>> data loss on the broker, ie, a previously acknowledged message no longer
>>> exists. For most part, this should only occur in rare situations
>>> (simultaneous power outages, multiple disk losses, software bugs
>>> resulting in data corruption, etc.).
>>>
>>>
>>> -Matthias
>>>
>>> On 9/29/17 7:55 AM, Damian Guy wrote:
>>> > You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
>>> >
>>> > Properties props = new Properties();
>>> > props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
>>> > ...
>>> >
>>> > On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com>
>>> wrote:
>>> >
>>> >> I guess once stream app are enabled exactly-once, producer
>>> idempotence get
>>> >> enabled by default and so do the retries. I guess producer retries are
>>> >> managed internally and not exposed through streamconfig.
>>> >>
>>> >> https://kafka.apache.org/0110/documentation/#streamsconfigs
>>> >>
>>> >> -Sameer.
>>> >>
>>> >> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <
>>> matthias@confluent.io>
>>> >> wrote:
>>> >>
>>> >>> An OutOfOrderSequenceException should only occur if a idempotent
>>> >>> producer gets out of sync with the broker. If you set
>>> >>> `enable.idempotence = true` on your producer, you might want to set
>>> >>> `retries = Integer.MAX_VALUE`.
>>> >>>
>>> >>> -Matthias
>>> >>>
>>> >>> On 9/26/17 11:30 PM, Sameer Kumar wrote:
>>> >>>> Hi,
>>> >>>>
>>> >>>> I again received this exception while running my streams app. I am
>>> >> using
>>> >>>> Kafka 11.0.1. After restarting my app, this error got fixed.
>>> >>>>
>>> >>>> I guess this might be due to bad network. Any pointers. Any config
>>> >>>> wherein I can configure it for retries.
>>> >>>>
>>> >>>> Exception trace is attached.
>>> >>>>
>>> >>>> Regards,
>>> >>>> -Sameer.
>>> >>>
>>> >>>
>>> >>
>>> >
>>>
>>>
>>
>
Re: out of order sequence number in exactly once streams
Posted by Sameer Kumar <sa...@gmail.com>.
I had enabled eos through streams config and as explained in the
documentation, I have not added anything else other than following config.
streamsConfiguration.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
StreamsConfig.EXACTLY_ONCE);
As explained by you, I think producer idempotence and retries would be
automatically picked up. I was thinking if there is someway i can print the
configs picked up the streams app for e.g. the current retries.
Do i need to add any other configuration.
-Sameer.
On Tue, Oct 3, 2017 at 11:59 AM, Sameer Kumar <sa...@gmail.com>
wrote:
> I had enabled eos through streams config.
>
>
> On Fri, Sep 29, 2017 at 11:12 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> That's correct: If EOS is enabled, we enforce some producer configs:
>>
>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/
>> main/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688
>>
>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/
>> main/java/org/apache/kafka/streams/StreamsConfig.java#L691
>>
>> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/
>> main/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496
>>
>>
>> Note, that by default we set retries to Integer.MAX_VALUE but we do not
>> enforce this setting (as pointed out by Damian already). So you could
>> overwrite it with a smaller value (what is of course not recommended).
>>
>> I was not sure though, if you enabled EOS or just enabled idempotency
>> only for the producer -- what you can easily do by providing the
>> corresponding producer configs.
>>
>> If you did enable EOS in StreamsConfig, Producer will take care of
>> OutOfOrderSequenceException in general.
>>
>> However, there are scenarios for which Producer cannot handle it. For
>> those case, OutOfOrderSequenceException indicates that there has been
>> data loss on the broker, ie, a previously acknowledged message no longer
>> exists. For most part, this should only occur in rare situations
>> (simultaneous power outages, multiple disk losses, software bugs
>> resulting in data corruption, etc.).
>>
>>
>> -Matthias
>>
>> On 9/29/17 7:55 AM, Damian Guy wrote:
>> > You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
>> >
>> > Properties props = new Properties();
>> > props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
>> > ...
>> >
>> > On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com>
>> wrote:
>> >
>> >> I guess once stream app are enabled exactly-once, producer idempotence
>> get
>> >> enabled by default and so do the retries. I guess producer retries are
>> >> managed internally and not exposed through streamconfig.
>> >>
>> >> https://kafka.apache.org/0110/documentation/#streamsconfigs
>> >>
>> >> -Sameer.
>> >>
>> >> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <
>> matthias@confluent.io>
>> >> wrote:
>> >>
>> >>> An OutOfOrderSequenceException should only occur if a idempotent
>> >>> producer gets out of sync with the broker. If you set
>> >>> `enable.idempotence = true` on your producer, you might want to set
>> >>> `retries = Integer.MAX_VALUE`.
>> >>>
>> >>> -Matthias
>> >>>
>> >>> On 9/26/17 11:30 PM, Sameer Kumar wrote:
>> >>>> Hi,
>> >>>>
>> >>>> I again received this exception while running my streams app. I am
>> >> using
>> >>>> Kafka 11.0.1. After restarting my app, this error got fixed.
>> >>>>
>> >>>> I guess this might be due to bad network. Any pointers. Any config
>> >>>> wherein I can configure it for retries.
>> >>>>
>> >>>> Exception trace is attached.
>> >>>>
>> >>>> Regards,
>> >>>> -Sameer.
>> >>>
>> >>>
>> >>
>> >
>>
>>
>
Re: out of order sequence number in exactly once streams
Posted by Sameer Kumar <sa...@gmail.com>.
I had enabled eos through streams config.
On Fri, Sep 29, 2017 at 11:12 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> That's correct: If EOS is enabled, we enforce some producer configs:
>
> https://github.com/apache/kafka/blob/0.11.0.1/streams/
> src/main/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688
>
> https://github.com/apache/kafka/blob/0.11.0.1/streams/
> src/main/java/org/apache/kafka/streams/StreamsConfig.java#L691
>
> https://github.com/apache/kafka/blob/0.11.0.1/streams/
> src/main/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496
>
>
> Note, that by default we set retries to Integer.MAX_VALUE but we do not
> enforce this setting (as pointed out by Damian already). So you could
> overwrite it with a smaller value (what is of course not recommended).
>
> I was not sure though, if you enabled EOS or just enabled idempotency
> only for the producer -- what you can easily do by providing the
> corresponding producer configs.
>
> If you did enable EOS in StreamsConfig, Producer will take care of
> OutOfOrderSequenceException in general.
>
> However, there are scenarios for which Producer cannot handle it. For
> those case, OutOfOrderSequenceException indicates that there has been
> data loss on the broker, ie, a previously acknowledged message no longer
> exists. For most part, this should only occur in rare situations
> (simultaneous power outages, multiple disk losses, software bugs
> resulting in data corruption, etc.).
>
>
> -Matthias
>
> On 9/29/17 7:55 AM, Damian Guy wrote:
> > You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
> >
> > Properties props = new Properties();
> > props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
> > ...
> >
> > On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com>
> wrote:
> >
> >> I guess once stream app are enabled exactly-once, producer idempotence
> get
> >> enabled by default and so do the retries. I guess producer retries are
> >> managed internally and not exposed through streamconfig.
> >>
> >> https://kafka.apache.org/0110/documentation/#streamsconfigs
> >>
> >> -Sameer.
> >>
> >> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <
> matthias@confluent.io>
> >> wrote:
> >>
> >>> An OutOfOrderSequenceException should only occur if a idempotent
> >>> producer gets out of sync with the broker. If you set
> >>> `enable.idempotence = true` on your producer, you might want to set
> >>> `retries = Integer.MAX_VALUE`.
> >>>
> >>> -Matthias
> >>>
> >>> On 9/26/17 11:30 PM, Sameer Kumar wrote:
> >>>> Hi,
> >>>>
> >>>> I again received this exception while running my streams app. I am
> >> using
> >>>> Kafka 11.0.1. After restarting my app, this error got fixed.
> >>>>
> >>>> I guess this might be due to bad network. Any pointers. Any config
> >>>> wherein I can configure it for retries.
> >>>>
> >>>> Exception trace is attached.
> >>>>
> >>>> Regards,
> >>>> -Sameer.
> >>>
> >>>
> >>
> >
>
>
Re: out of order sequence number in exactly once streams
Posted by "Matthias J. Sax" <ma...@confluent.io>.
That's correct: If EOS is enabled, we enforce some producer configs:
https://github.com/apache/kafka/blob/0.11.0.1/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688
https://github.com/apache/kafka/blob/0.11.0.1/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java#L691
https://github.com/apache/kafka/blob/0.11.0.1/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496
Note, that by default we set retries to Integer.MAX_VALUE but we do not
enforce this setting (as pointed out by Damian already). So you could
overwrite it with a smaller value (what is of course not recommended).
I was not sure though, if you enabled EOS or just enabled idempotency
only for the producer -- what you can easily do by providing the
corresponding producer configs.
If you did enable EOS in StreamsConfig, Producer will take care of
OutOfOrderSequenceException in general.
However, there are scenarios for which Producer cannot handle it. For
those case, OutOfOrderSequenceException indicates that there has been
data loss on the broker, ie, a previously acknowledged message no longer
exists. For most part, this should only occur in rare situations
(simultaneous power outages, multiple disk losses, software bugs
resulting in data corruption, etc.).
-Matthias
On 9/29/17 7:55 AM, Damian Guy wrote:
> You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
>
> Properties props = new Properties();
> props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
> ...
>
> On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com> wrote:
>
>> I guess once stream app are enabled exactly-once, producer idempotence get
>> enabled by default and so do the retries. I guess producer retries are
>> managed internally and not exposed through streamconfig.
>>
>> https://kafka.apache.org/0110/documentation/#streamsconfigs
>>
>> -Sameer.
>>
>> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <ma...@confluent.io>
>> wrote:
>>
>>> An OutOfOrderSequenceException should only occur if a idempotent
>>> producer gets out of sync with the broker. If you set
>>> `enable.idempotence = true` on your producer, you might want to set
>>> `retries = Integer.MAX_VALUE`.
>>>
>>> -Matthias
>>>
>>> On 9/26/17 11:30 PM, Sameer Kumar wrote:
>>>> Hi,
>>>>
>>>> I again received this exception while running my streams app. I am
>> using
>>>> Kafka 11.0.1. After restarting my app, this error got fixed.
>>>>
>>>> I guess this might be due to bad network. Any pointers. Any config
>>>> wherein I can configure it for retries.
>>>>
>>>> Exception trace is attached.
>>>>
>>>> Regards,
>>>> -Sameer.
>>>
>>>
>>
>
Re: out of order sequence number in exactly once streams
Posted by Damian Guy <da...@gmail.com>.
You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e,
Properties props = new Properties();
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
...
On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sa...@gmail.com> wrote:
> I guess once stream app are enabled exactly-once, producer idempotence get
> enabled by default and so do the retries. I guess producer retries are
> managed internally and not exposed through streamconfig.
>
> https://kafka.apache.org/0110/documentation/#streamsconfigs
>
> -Sameer.
>
> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > An OutOfOrderSequenceException should only occur if a idempotent
> > producer gets out of sync with the broker. If you set
> > `enable.idempotence = true` on your producer, you might want to set
> > `retries = Integer.MAX_VALUE`.
> >
> > -Matthias
> >
> > On 9/26/17 11:30 PM, Sameer Kumar wrote:
> > > Hi,
> > >
> > > I again received this exception while running my streams app. I am
> using
> > > Kafka 11.0.1. After restarting my app, this error got fixed.
> > >
> > > I guess this might be due to bad network. Any pointers. Any config
> > > wherein I can configure it for retries.
> > >
> > > Exception trace is attached.
> > >
> > > Regards,
> > > -Sameer.
> >
> >
>
Re: out of order sequence number in exactly once streams
Posted by Sameer Kumar <sa...@gmail.com>.
I guess once stream app are enabled exactly-once, producer idempotence get
enabled by default and so do the retries. I guess producer retries are
managed internally and not exposed through streamconfig.
https://kafka.apache.org/0110/documentation/#streamsconfigs
-Sameer.
On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> An OutOfOrderSequenceException should only occur if a idempotent
> producer gets out of sync with the broker. If you set
> `enable.idempotence = true` on your producer, you might want to set
> `retries = Integer.MAX_VALUE`.
>
> -Matthias
>
> On 9/26/17 11:30 PM, Sameer Kumar wrote:
> > Hi,
> >
> > I again received this exception while running my streams app. I am using
> > Kafka 11.0.1. After restarting my app, this error got fixed.
> >
> > I guess this might be due to bad network. Any pointers. Any config
> > wherein I can configure it for retries.
> >
> > Exception trace is attached.
> >
> > Regards,
> > -Sameer.
>
>
Re: out of order sequence number in exactly once streams
Posted by "Matthias J. Sax" <ma...@confluent.io>.
An OutOfOrderSequenceException should only occur if a idempotent
producer gets out of sync with the broker. If you set
`enable.idempotence = true` on your producer, you might want to set
`retries = Integer.MAX_VALUE`.
-Matthias
On 9/26/17 11:30 PM, Sameer Kumar wrote:
> Hi,
>
> I again received this exception while running my streams app. I am using
> Kafka 11.0.1. After restarting my app, this error got fixed.
>
> I guess this might be due to bad network. Any pointers. Any config
> wherein I can configure it for retries.
>
> Exception trace is attached.
>
> Regards,
> -Sameer.