You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Matthias J. Sax" <ma...@confluent.io> on 2018/09/07 04:52:49 UTC
Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto
Topic Creation
What is the status of this KIP?
I think you can start a VOTE Dhruvil.
-Matthias
On 8/23/18 9:52 AM, Ismael Juma wrote:
> Yeah, the reason why we want to deprecate the auto create functionality is
> that it happens when a metadata request is done instead of when a write
> operation happens. So, there's no reason to differentiate between the two.
>
> Ismael
>
> On Thu, Aug 23, 2018 at 8:16 AM Andrew Otto <ot...@wikimedia.org> wrote:
>
>> Ah, I just realized that as proposed this is only for the Java consumer
>> client, correct? Would it be possible to make this a broker config, like
>> the current one? Something like:
>>
>> auto.create.topics.enable=true # allow both producer and consumer to create
>> auto.create.topics.enable=consumer # allow only consumer to create
>> auto.create.topics.enable=producer # allow only producer to create
>> auto.create.topics.enable=false # deny any auto topic creation
>>
>> Perhaps the broker doesn’t differentiate between the type of client
>> connection. If not, I guess this wouldn’t be possible.
>>
>>
>>
>> On Thu, Aug 23, 2018 at 11:08 AM Andrew Otto <ot...@wikimedia.org> wrote:
>>
>>> Yup :)
>>>
>>> On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma <is...@juma.me.uk> wrote:
>>>
>>>> Andrew, one question: you are relying on auto topic creation for the
>>>> producer and that's why you can't just disable it?
>>>>
>>>> On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma <is...@juma.me.uk> wrote:
>>>>
>>>>> Thanks for sharing Andrew!
>>>>>
>>>>> Ismael
>>>>>
>>>>> On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto <ot...@wikimedia.org>
>> wrote:
>>>>>
>>>>>> We recently had a pretty serious Kafka outage
>>>>>> <
>>>>>>
>>>>
>> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary
>>>>>>>
>>>>>> caused by a bug in one of our consumers that caused it to create new
>>>>>> topics
>>>>>> in an infinite loop AKA a topic bomb! Having consumers restricted
>> from
>>>>>> creating topics would have prevented this for us.
>>>>>>
>>>>>> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma <is...@juma.me.uk>
>> wrote:
>>>>>>
>>>>>>> Generally, I think positive configs (`allow` instead of `suppress`)
>>>> are
>>>>>>> easier to understand.
>>>>>>>
>>>>>>> Ismael
>>>>>>>
>>>>>>> On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <
>>>> matthias@confluent.io
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the summary!
>>>>>>>>
>>>>>>>> We might want to add a diagram/table to the docs when we add this
>>>>>>>> feature (with whatever config name we choose) to explain how
>> broker
>>>>>>>> config `auto.create.topics.enable` and the consumer config work
>>>>>> together.
>>>>>>>>
>>>>>>>> I think both options are equally easy to understand. "allow"
>> means
>>>>>>>> follow the broker config, while "suppress" implies ignore the
>>>> broker
>>>>>>>> config and don't auto-create.
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/22/18 10:36 PM, Dhruvil Shah wrote:
>>>>>>>>> *"suppress" is the opposite of "allow", so
>>>>>>>>> setting suppress.auto.create.topics=false would mean that we do
>>>>>> _not_
>>>>>>>> allow
>>>>>>>>> auto topic creation; when set to true, the server configuration
>>>> will
>>>>>>>>> determine whether we allow automatic creation or not.*
>>>>>>>>>
>>>>>>>>> Sorry, I meant suppress.auto.create.topics=true above to
>> disallow
>>>>>> auto
>>>>>>>>> topic creation.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <
>>>> dhruvil@confluent.io
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> To be clear, we will allow auto topic creation only when
>> server
>>>>>> config
>>>>>>>>>> auto.create.topics.enable=true and consumer config
>>>>>>>>>> allow.auto.create.topics=true; when either is false, we would
>>>> not
>>>>>>> create
>>>>>>>>>> the topic if it does not exist.
>>>>>>>>>>
>>>>>>>>>> "suppress" is the opposite of "allow", so
>>>>>>>>>> setting suppress.auto.create.topics=false would mean that we
>> do
>>>>>> _not_
>>>>>>>> allow
>>>>>>>>>> auto topic creation; when set to true, the server
>> configuration
>>>>>> will
>>>>>>>>>> determine whether we allow automatic creation or not.
>>>>>>>>>>
>>>>>>>>>> I think "allow" is easier to understand but I am open to
>>>>>> suggestions.
>>>>>>>>>>
>>>>>>>>>> - Dhruvil
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
>>>>>>>>>> brandon.kirchner@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> “allow=false” seems a bit more intuitive to me than
>>>>>> “suppress=false”
>>>>>>>>>>>
>>>>>>>>>>> Brandon
>>>>>>>>>>>
>>>>>>>>>>>> On Aug 22, 2018, at 8:48 PM, Ted Yu <yu...@gmail.com>
>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> We may also consider :
>>>>>>>>>>>>
>>>>>>>>>>>> "suppress.auto.topic.creation"
>>>>>>>>>>>>
>>>>>>>>>>>> or
>>>>>>>>>>>>
>>>>>>>>>>>> "allow.auto.topic.creation"
>>>>>>>>>>>>
>>>>>>>>>>>> w.r.t. suppress or allow, I don't have strong opinion
>> either.
>>>>>> It's
>>>>>>>> just
>>>>>>>>>>> a
>>>>>>>>>>>> matter of choosing the proper default value.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah <
>>>>>> dhruvil@confluent.io
>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Matthias,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do you mean something like "suppress.auto.create.topic"? I
>> am
>>>>>>> leaning
>>>>>>>>>>> a bit
>>>>>>>>>>>>> towards "allow.auto.create.topics" but I don't have a
>> strong
>>>>>>>> preference
>>>>>>>>>>>>> either. Let's wait to hear if anyone else has an opinion on
>>>>>> this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Dhruvil
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax <
>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the KIP Dhruvil!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree with Jason's comment. An alternative might be to
>> use
>>>>>>>>>>> "suppress"
>>>>>>>>>>>>>> what would revert the logic of "allow". Not sure which one
>>>> is
>>>>>> more
>>>>>>>>>>>>>> intuitive and I am fine with both (no personal
>> preference).
>>>>>> Just
>>>>>>>>>>> wanted
>>>>>>>>>>>>>> to mention it as an alternative.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Don't have any further comments/question so far.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
>>>>>>>>>>>>>>> Hey Dhruvil,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would suggest using the verb "allow" rather than
>> "enable.
>>>>>> The
>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>> cannot enable auto topic creation because it is
>> configured
>>>> on
>>>>>> the
>>>>>>>>>>>>> broker.
>>>>>>>>>>>>>>> All it can do is prevent it from happening if it is
>>>> enabled.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Jason
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah <
>>>>>>>> dhruvil@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would like to start discussion on KIP-361 that
>> proposes
>>>> we
>>>>>>> add a
>>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>>> configuration to disable auto topic creation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Link to the KIP:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
>>>>>>>>>>>>>>>> Configuration+to+Disable+Auto+Topic+Creation
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Suggestions and feedback are welcome!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Dhruvil
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto
Topic Creation
Posted by Dhruvil Shah <dh...@confluent.io>.
Hi all,
I updated the KIP with the discussion from this thread. I left the warning
message and deprecation out for now because we require the configuration be
set to true (i.e. auto topic creation allowed) when using brokers older
than 0.11.0.
If there is no more feedback, I will start a VOTE thread tomorrow.
Thanks,
Dhruvil
On Thu, Sep 6, 2018 at 9:52 PM Matthias J. Sax <ma...@confluent.io>
wrote:
> What is the status of this KIP?
>
> I think you can start a VOTE Dhruvil.
>
>
> -Matthias
>
> On 8/23/18 9:52 AM, Ismael Juma wrote:
> > Yeah, the reason why we want to deprecate the auto create functionality
> is
> > that it happens when a metadata request is done instead of when a write
> > operation happens. So, there's no reason to differentiate between the
> two.
> >
> > Ismael
> >
> > On Thu, Aug 23, 2018 at 8:16 AM Andrew Otto <ot...@wikimedia.org> wrote:
> >
> >> Ah, I just realized that as proposed this is only for the Java consumer
> >> client, correct? Would it be possible to make this a broker config,
> like
> >> the current one? Something like:
> >>
> >> auto.create.topics.enable=true # allow both producer and consumer to
> create
> >> auto.create.topics.enable=consumer # allow only consumer to create
> >> auto.create.topics.enable=producer # allow only producer to create
> >> auto.create.topics.enable=false # deny any auto topic creation
> >>
> >> Perhaps the broker doesn’t differentiate between the type of client
> >> connection. If not, I guess this wouldn’t be possible.
> >>
> >>
> >>
> >> On Thu, Aug 23, 2018 at 11:08 AM Andrew Otto <ot...@wikimedia.org>
> wrote:
> >>
> >>> Yup :)
> >>>
> >>> On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma <is...@juma.me.uk>
> wrote:
> >>>
> >>>> Andrew, one question: you are relying on auto topic creation for the
> >>>> producer and that's why you can't just disable it?
> >>>>
> >>>> On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma <is...@juma.me.uk>
> wrote:
> >>>>
> >>>>> Thanks for sharing Andrew!
> >>>>>
> >>>>> Ismael
> >>>>>
> >>>>> On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto <ot...@wikimedia.org>
> >> wrote:
> >>>>>
> >>>>>> We recently had a pretty serious Kafka outage
> >>>>>> <
> >>>>>>
> >>>>
> >>
> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary
> >>>>>>>
> >>>>>> caused by a bug in one of our consumers that caused it to create new
> >>>>>> topics
> >>>>>> in an infinite loop AKA a topic bomb! Having consumers restricted
> >> from
> >>>>>> creating topics would have prevented this for us.
> >>>>>>
> >>>>>> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma <is...@juma.me.uk>
> >> wrote:
> >>>>>>
> >>>>>>> Generally, I think positive configs (`allow` instead of `suppress`)
> >>>> are
> >>>>>>> easier to understand.
> >>>>>>>
> >>>>>>> Ismael
> >>>>>>>
> >>>>>>> On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <
> >>>> matthias@confluent.io
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks for the summary!
> >>>>>>>>
> >>>>>>>> We might want to add a diagram/table to the docs when we add this
> >>>>>>>> feature (with whatever config name we choose) to explain how
> >> broker
> >>>>>>>> config `auto.create.topics.enable` and the consumer config work
> >>>>>> together.
> >>>>>>>>
> >>>>>>>> I think both options are equally easy to understand. "allow"
> >> means
> >>>>>>>> follow the broker config, while "suppress" implies ignore the
> >>>> broker
> >>>>>>>> config and don't auto-create.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> >>>>>>>>> *"suppress" is the opposite of "allow", so
> >>>>>>>>> setting suppress.auto.create.topics=false would mean that we do
> >>>>>> _not_
> >>>>>>>> allow
> >>>>>>>>> auto topic creation; when set to true, the server configuration
> >>>> will
> >>>>>>>>> determine whether we allow automatic creation or not.*
> >>>>>>>>>
> >>>>>>>>> Sorry, I meant suppress.auto.create.topics=true above to
> >> disallow
> >>>>>> auto
> >>>>>>>>> topic creation.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <
> >>>> dhruvil@confluent.io
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> To be clear, we will allow auto topic creation only when
> >> server
> >>>>>> config
> >>>>>>>>>> auto.create.topics.enable=true and consumer config
> >>>>>>>>>> allow.auto.create.topics=true; when either is false, we would
> >>>> not
> >>>>>>> create
> >>>>>>>>>> the topic if it does not exist.
> >>>>>>>>>>
> >>>>>>>>>> "suppress" is the opposite of "allow", so
> >>>>>>>>>> setting suppress.auto.create.topics=false would mean that we
> >> do
> >>>>>> _not_
> >>>>>>>> allow
> >>>>>>>>>> auto topic creation; when set to true, the server
> >> configuration
> >>>>>> will
> >>>>>>>>>> determine whether we allow automatic creation or not.
> >>>>>>>>>>
> >>>>>>>>>> I think "allow" is easier to understand but I am open to
> >>>>>> suggestions.
> >>>>>>>>>>
> >>>>>>>>>> - Dhruvil
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
> >>>>>>>>>> brandon.kirchner@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> “allow=false” seems a bit more intuitive to me than
> >>>>>> “suppress=false”
> >>>>>>>>>>>
> >>>>>>>>>>> Brandon
> >>>>>>>>>>>
> >>>>>>>>>>>> On Aug 22, 2018, at 8:48 PM, Ted Yu <yu...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> We may also consider :
> >>>>>>>>>>>>
> >>>>>>>>>>>> "suppress.auto.topic.creation"
> >>>>>>>>>>>>
> >>>>>>>>>>>> or
> >>>>>>>>>>>>
> >>>>>>>>>>>> "allow.auto.topic.creation"
> >>>>>>>>>>>>
> >>>>>>>>>>>> w.r.t. suppress or allow, I don't have strong opinion
> >> either.
> >>>>>> It's
> >>>>>>>> just
> >>>>>>>>>>> a
> >>>>>>>>>>>> matter of choosing the proper default value.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah <
> >>>>>> dhruvil@confluent.io
> >>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Matthias,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Do you mean something like "suppress.auto.create.topic"? I
> >> am
> >>>>>>> leaning
> >>>>>>>>>>> a bit
> >>>>>>>>>>>>> towards "allow.auto.create.topics" but I don't have a
> >> strong
> >>>>>>>> preference
> >>>>>>>>>>>>> either. Let's wait to hear if anyone else has an opinion on
> >>>>>> this.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Dhruvil
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax <
> >>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the KIP Dhruvil!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I agree with Jason's comment. An alternative might be to
> >> use
> >>>>>>>>>>> "suppress"
> >>>>>>>>>>>>>> what would revert the logic of "allow". Not sure which one
> >>>> is
> >>>>>> more
> >>>>>>>>>>>>>> intuitive and I am fine with both (no personal
> >> preference).
> >>>>>> Just
> >>>>>>>>>>> wanted
> >>>>>>>>>>>>>> to mention it as an alternative.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Don't have any further comments/question so far.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
> >>>>>>>>>>>>>>> Hey Dhruvil,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I would suggest using the verb "allow" rather than
> >> "enable.
> >>>>>> The
> >>>>>>>>>>>>> consumer
> >>>>>>>>>>>>>>> cannot enable auto topic creation because it is
> >> configured
> >>>> on
> >>>>>> the
> >>>>>>>>>>>>> broker.
> >>>>>>>>>>>>>>> All it can do is prevent it from happening if it is
> >>>> enabled.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Jason
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah <
> >>>>>>>> dhruvil@confluent.io>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I would like to start discussion on KIP-361 that
> >> proposes
> >>>> we
> >>>>>>> add a
> >>>>>>>>>>>>>> consumer
> >>>>>>>>>>>>>>>> configuration to disable auto topic creation.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Link to the KIP:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
> >>>>>>>>>>>>>>>> Configuration+to+Disable+Auto+Topic+Creation
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Suggestions and feedback are welcome!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> Dhruvil
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
>