You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dmitriy Vsekhvalnov <dv...@gmail.com> on 2017/10/10 12:24:00 UTC
kafka-streams dying if can't create internal topics
Hi all,
still doing disaster testing with Kafka cluster, when crashing several
brokers at once sometimes we observe exception in kafka-stream app about
inability to create internal topics:
[WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
[Could not create internal topics: Found only 2 brokers, but replication
factor is 3. Decrease replication factor for internal topics via
StreamsConfig parameter "replication.factor" or add more brokers to your
cluster. Retry #2]
[WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
[Could not create internal topics: Found only 2 brokers, but replication
factor is 3. Decrease replication factor for internal topics via
StreamsConfig parameter "replication.factor" or add more brokers to your
cluster. Retry #3]
[WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
[Could not create internal topics: Found only 2 brokers, but replication
factor is 3. Decrease replication factor for internal topics via
StreamsConfig parameter "replication.factor" or add more brokers to your
cluster. Retry #4]
[INFO ] [org.apache.kafka.streams.processor.internals.StreamThread]
[stream-thread [Shutting down]
The problem is that number of retries seems to be hardcoded to 5.
From InternalTopicManager.MAX_TOPIC_READY_TRY constant.
Any way to make to configurable? It's really not nice that app is shutting
down, instead of just re-trying (potentially with exponential backoff)
until all broker are available back.
Should we open feature/issue request?
Thank you.
Re: kafka-streams dying if can't create internal topics
Posted by Dmitriy Vsekhvalnov <dv...@gmail.com>.
Thanks Matthias,
here is issue: https://issues.apache.org/jira/browse/KAFKA-6047
On Tue, Oct 10, 2017 at 8:50 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> https://issues.apache.org/jira/browse/KAFKA-1?jql=project%20%3D%20KAFKA
>
> Or just put "apache kafka jira" into your favorite search engine...
>
>
> -Matthias
>
>
> On 10/10/17 10:48 AM, Dmitriy Vsekhvalnov wrote:
> > Hi Matthias,
> >
> > thanks. Would you mind point me to correct Jira URL where i can file an
> > issue?
> >
> > Thanks again.
> >
> > On Tue, Oct 10, 2017 at 8:38 PM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Yes, please file a Jira. We need to fix this. Thanks a lot!
> >>
> >> -Matthias
> >>
> >> On 10/10/17 5:24 AM, Dmitriy Vsekhvalnov wrote:
> >>> Hi all,
> >>>
> >>> still doing disaster testing with Kafka cluster, when crashing several
> >>> brokers at once sometimes we observe exception in kafka-stream app
> about
> >>> inability to create internal topics:
> >>>
> >>> [WARN ] [org.apache.kafka.streams.processor.internals.
> >> InternalTopicManager]
> >>> [Could not create internal topics: Found only 2 brokers, but
> replication
> >>> factor is 3. Decrease replication factor for internal topics via
> >>> StreamsConfig parameter "replication.factor" or add more brokers to
> your
> >>> cluster. Retry #2]
> >>> [WARN ] [org.apache.kafka.streams.processor.internals.
> >> InternalTopicManager]
> >>> [Could not create internal topics: Found only 2 brokers, but
> replication
> >>> factor is 3. Decrease replication factor for internal topics via
> >>> StreamsConfig parameter "replication.factor" or add more brokers to
> your
> >>> cluster. Retry #3]
> >>> [WARN ] [org.apache.kafka.streams.processor.internals.
> >> InternalTopicManager]
> >>> [Could not create internal topics: Found only 2 brokers, but
> replication
> >>> factor is 3. Decrease replication factor for internal topics via
> >>> StreamsConfig parameter "replication.factor" or add more brokers to
> your
> >>> cluster. Retry #4]
> >>> [INFO ] [org.apache.kafka.streams.processor.internals.StreamThread]
> >>> [stream-thread [Shutting down]
> >>>
> >>> The problem is that number of retries seems to be hardcoded to 5.
> >>> From InternalTopicManager.MAX_TOPIC_READY_TRY constant.
> >>>
> >>> Any way to make to configurable? It's really not nice that app is
> >> shutting
> >>> down, instead of just re-trying (potentially with exponential backoff)
> >>> until all broker are available back.
> >>>
> >>> Should we open feature/issue request?
> >>>
> >>> Thank you.
> >>>
> >>
> >>
> >
>
>
Re: kafka-streams dying if can't create internal topics
Posted by "Matthias J. Sax" <ma...@confluent.io>.
https://issues.apache.org/jira/browse/KAFKA-1?jql=project%20%3D%20KAFKA
Or just put "apache kafka jira" into your favorite search engine...
-Matthias
On 10/10/17 10:48 AM, Dmitriy Vsekhvalnov wrote:
> Hi Matthias,
>
> thanks. Would you mind point me to correct Jira URL where i can file an
> issue?
>
> Thanks again.
>
> On Tue, Oct 10, 2017 at 8:38 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Yes, please file a Jira. We need to fix this. Thanks a lot!
>>
>> -Matthias
>>
>> On 10/10/17 5:24 AM, Dmitriy Vsekhvalnov wrote:
>>> Hi all,
>>>
>>> still doing disaster testing with Kafka cluster, when crashing several
>>> brokers at once sometimes we observe exception in kafka-stream app about
>>> inability to create internal topics:
>>>
>>> [WARN ] [org.apache.kafka.streams.processor.internals.
>> InternalTopicManager]
>>> [Could not create internal topics: Found only 2 brokers, but replication
>>> factor is 3. Decrease replication factor for internal topics via
>>> StreamsConfig parameter "replication.factor" or add more brokers to your
>>> cluster. Retry #2]
>>> [WARN ] [org.apache.kafka.streams.processor.internals.
>> InternalTopicManager]
>>> [Could not create internal topics: Found only 2 brokers, but replication
>>> factor is 3. Decrease replication factor for internal topics via
>>> StreamsConfig parameter "replication.factor" or add more brokers to your
>>> cluster. Retry #3]
>>> [WARN ] [org.apache.kafka.streams.processor.internals.
>> InternalTopicManager]
>>> [Could not create internal topics: Found only 2 brokers, but replication
>>> factor is 3. Decrease replication factor for internal topics via
>>> StreamsConfig parameter "replication.factor" or add more brokers to your
>>> cluster. Retry #4]
>>> [INFO ] [org.apache.kafka.streams.processor.internals.StreamThread]
>>> [stream-thread [Shutting down]
>>>
>>> The problem is that number of retries seems to be hardcoded to 5.
>>> From InternalTopicManager.MAX_TOPIC_READY_TRY constant.
>>>
>>> Any way to make to configurable? It's really not nice that app is
>> shutting
>>> down, instead of just re-trying (potentially with exponential backoff)
>>> until all broker are available back.
>>>
>>> Should we open feature/issue request?
>>>
>>> Thank you.
>>>
>>
>>
>
Re: kafka-streams dying if can't create internal topics
Posted by Dmitriy Vsekhvalnov <dv...@gmail.com>.
Hi Matthias,
thanks. Would you mind point me to correct Jira URL where i can file an
issue?
Thanks again.
On Tue, Oct 10, 2017 at 8:38 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Yes, please file a Jira. We need to fix this. Thanks a lot!
>
> -Matthias
>
> On 10/10/17 5:24 AM, Dmitriy Vsekhvalnov wrote:
> > Hi all,
> >
> > still doing disaster testing with Kafka cluster, when crashing several
> > brokers at once sometimes we observe exception in kafka-stream app about
> > inability to create internal topics:
> >
> > [WARN ] [org.apache.kafka.streams.processor.internals.
> InternalTopicManager]
> > [Could not create internal topics: Found only 2 brokers, but replication
> > factor is 3. Decrease replication factor for internal topics via
> > StreamsConfig parameter "replication.factor" or add more brokers to your
> > cluster. Retry #2]
> > [WARN ] [org.apache.kafka.streams.processor.internals.
> InternalTopicManager]
> > [Could not create internal topics: Found only 2 brokers, but replication
> > factor is 3. Decrease replication factor for internal topics via
> > StreamsConfig parameter "replication.factor" or add more brokers to your
> > cluster. Retry #3]
> > [WARN ] [org.apache.kafka.streams.processor.internals.
> InternalTopicManager]
> > [Could not create internal topics: Found only 2 brokers, but replication
> > factor is 3. Decrease replication factor for internal topics via
> > StreamsConfig parameter "replication.factor" or add more brokers to your
> > cluster. Retry #4]
> > [INFO ] [org.apache.kafka.streams.processor.internals.StreamThread]
> > [stream-thread [Shutting down]
> >
> > The problem is that number of retries seems to be hardcoded to 5.
> > From InternalTopicManager.MAX_TOPIC_READY_TRY constant.
> >
> > Any way to make to configurable? It's really not nice that app is
> shutting
> > down, instead of just re-trying (potentially with exponential backoff)
> > until all broker are available back.
> >
> > Should we open feature/issue request?
> >
> > Thank you.
> >
>
>
Re: kafka-streams dying if can't create internal topics
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Yes, please file a Jira. We need to fix this. Thanks a lot!
-Matthias
On 10/10/17 5:24 AM, Dmitriy Vsekhvalnov wrote:
> Hi all,
>
> still doing disaster testing with Kafka cluster, when crashing several
> brokers at once sometimes we observe exception in kafka-stream app about
> inability to create internal topics:
>
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
> [Could not create internal topics: Found only 2 brokers, but replication
> factor is 3. Decrease replication factor for internal topics via
> StreamsConfig parameter "replication.factor" or add more brokers to your
> cluster. Retry #2]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
> [Could not create internal topics: Found only 2 brokers, but replication
> factor is 3. Decrease replication factor for internal topics via
> StreamsConfig parameter "replication.factor" or add more brokers to your
> cluster. Retry #3]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
> [Could not create internal topics: Found only 2 brokers, but replication
> factor is 3. Decrease replication factor for internal topics via
> StreamsConfig parameter "replication.factor" or add more brokers to your
> cluster. Retry #4]
> [INFO ] [org.apache.kafka.streams.processor.internals.StreamThread]
> [stream-thread [Shutting down]
>
> The problem is that number of retries seems to be hardcoded to 5.
> From InternalTopicManager.MAX_TOPIC_READY_TRY constant.
>
> Any way to make to configurable? It's really not nice that app is shutting
> down, instead of just re-trying (potentially with exponential backoff)
> until all broker are available back.
>
> Should we open feature/issue request?
>
> Thank you.
>