You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Justine Olshan <jo...@confluent.io> on 2019/08/06 00:24:18 UTC

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi all,
I made some changes to the KIP. Hopefully this configuration change will
make things a little clearer.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer

Please let me know if you have any feedback or questions!

Thank you,
Justine

On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org> wrote:

> Hi Mickael,
>
> I think you bring up a good point.  It would be better if we didn't ever
> have to set up client-side configuration for this feature, and KIP-464
> would let us skip this entirely.
>
> On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > Hi Mickael,
> > I agree that KIP-464 works on newer brokers, but I was a bit worried how
> > things would play out on older brokers that* do not *have KIP 464
> included.
> > Is it enough to throw an error in this case when producer configs are not
> > specified?
>
> I think the right thing to do would be to log an error message in the
> client.  We will need to have that capability in any case, to cover
> scenarios like the client trying to auto-create a topic that they don't
> have permission to create.  Or a client trying to create a topic on a
> broker so old that CreateTopicsRequest is not supported.
>
> The big downside to relying on KIP-464 is that it is a very recent feature
> -- so recent that it hasn't even made its way to any official Apache
> release.  It's scheduled for the upcoming 2.4 release in a few months.
>
> So if you view this KIP as a step towards removing broker-side
> auto-create, you might want to support older brokers just to accelerate
> adoption, and hasten the day when we can finally flip broker-side
> auto-create to off (or even remove it entirely).
>
> I have to agree, though, that having client-side configurations for number
> of partitions and replication factor is messy.  Maybe it would be worth it
> to restrict support to post-KIP-464 brokers, if we could avoid creating
> more configs.
>
> best,
> Colin
>
>
> > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <mickael.maison@gmail.com
> >
> > wrote:
> >
> > > Hi Justine,
> > >
> > > We can rely on KIP-464 which allows to omit the partition count or
> > > replication factor when creating a topic. In that case, the broker
> > > defaults are used.
> > >
> > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > > wrote:
> > > >
> > > > Michael,
> > > > That makes sense to me!
> > > > To clarify, in the current state of the KIP, the producer does not
> rely
> > > on
> > > > the broker to autocreate--if the broker's config is disabled, then
> the
> > > > producer can autocreate on its own with a create topic request (the
> same
> > > > type of request the admin client uses).
> > > > However, if both configs are enabled, the broker will autocreate
> through
> > > a
> > > > metadata request before the producer gets a chance.
> > > > Of course, the way to avoid this, is to do as you suggested, and set
> the
> > > > "allow_auto_topic_creation" field to false.
> > > >
> > > > I think the only thing we need to be careful with in this setup is
> > > without
> > > > KIP 464, we can not use broker defaults for this topic. A user needs
> to
> > > > specify the number of partition and replication factor in the config.
> > > > An alternative to this is to have coded defaults for when these
> configs
> > > are
> > > > unspecified, but it is not immediately apparent what these defaults
> > > should
> > > > be.
> > > >
> > > > Thanks again for reading my KIP,
> > > > Justine
> > > >
> > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the response!
> > > > > In my opinion, it would be better if the producer did not rely at
> all
> > > > > on the broker auto create feature as this is what we're aiming to
> > > > > deprecate. When requesting metadata we can set the
> > > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > > creation. Then if the topic is not existing, send a
> > > > > CreateTopicRequest.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> jolshan@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > Currently the way it is implemented, the broker auto-creation
> > > > > configuration
> > > > > > takes precedence. The producer will not use the CreateTopics
> request.
> > > > > > (Technically it can--but the topic will already be created
> through
> > > the
> > > > > > broker, so it will never try to create the topic.)
> > > > > > It is possible to change this however, and I'd be happy to
> discuss
> > > the
> > > > > > benefits of this alternative.
> > > > > >
> > > > > > Thank you,
> > > > > > Justine
> > > > > >
> > > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > > > > mickael.maison@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Thanks for the KIP!
> > > > > > >
> > > > > > > In case auto creation is enabled on both the client and server,
> > > will
> > > > > > > the producer still use the AdminClient (CreateTopics request)
> to
> > > > > > > create topics? and not rely on the broker auto create.
> > > > > > > I'm guessing the answer is yes but can you make it explicit.
> > > > > > >
> > > > > > > Thank you
> > > > > > >
> > > > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > > jolshan@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > > Just a friendly reminder to take a look at this KIP if you
> have
> > > the
> > > > > time.
> > > > > > > >
> > > > > > > > I was thinking about broker vs. client default precedence,
> and I
> > > > > think it
> > > > > > > > makes sense to keep the broker as the default used when both
> > > > > client-side
> > > > > > > > and broker-side defaults are configured. The idea is that
> there
> > > > > would be
> > > > > > > > pretty clear documentation, and that many systems with
> > > configurations
> > > > > > > that
> > > > > > > > the client could not change would likely have the auto-create
> > > default
> > > > > > > off.
> > > > > > > > (In cloud for example).
> > > > > > > >
> > > > > > > > It also seems like in most cases, the consumer config
> > > > > > > > 'allow.auto.create.topics' was created to actually prevent
> the
> > > > > creation
> > > > > > > of
> > > > > > > > topics, so the loss of creation functionality will not be a
> big
> > > > > problem.
> > > > > > > >
> > > > > > > >  I'm happy to discuss any other compatibility problems or
> > > components
> > > > > of
> > > > > > > > this KIP.
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > > jolshan@confluent.io
> > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello all,
> > > > > > > > >
> > > > > > > > > I was looking at this KIP again, and there is a decision I
> made
> > > > > that I
> > > > > > > > > think is worth discussing.
> > > > > > > > >
> > > > > > > > > In the case where both the broker and producer's
> > > > > > > > > 'auto.create.topics.enable' are set to true, we have to
> choose
> > > > > either
> > > > > > > the
> > > > > > > > > broker configs or the producer configs for the replication
> > > > > > > > > factor/partitions.
> > > > > > > > >
> > > > > > > > > Currently, the decision is to have the broker defaults take
> > > > > precedence.
> > > > > > > > > (It is easier to do this in the implementation.) It also
> makes
> > > some
> > > > > > > sense
> > > > > > > > > for this behavior to take precedence since this behavior
> > > already
> > > > > > > occurs as
> > > > > > > > > the default.
> > > > > > > > >
> > > > > > > > > However, I was wondering if it would be odd for those who
> can
> > > only
> > > > > see
> > > > > > > the
> > > > > > > > > producer side to set configs for replication
> factor/partitions
> > > and
> > > > > see
> > > > > > > > > different results. Currently the documentation for the
> config
> > > > > states
> > > > > > > that
> > > > > > > > > the config values are only used when the broker config is
> not
> > > > > enabled,
> > > > > > > but
> > > > > > > > > this might not always be clear to the user.  Changing the
> code
> > > to
> > > > > have
> > > > > > > the
> > > > > > > > > producer's configurations take precedence is possible, but
> I
> > > was
> > > > > > > wondering
> > > > > > > > > what everyone thought.
> > > > > > > > >
> > > > > > > > > Thank you,
> > > > > > > > > Justine
> > > > > > > > >
> > > > > > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > > > > jolshan@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Just a quick update--
> > > > > > > > >>
> > > > > > > > >> It seems that enabling both the broker and producer
> configs
> > > works
> > > > > > > fine,
> > > > > > > > >> except that the broker configurations for partitions,
> > > replication
> > > > > > > factor
> > > > > > > > >> take precedence.
> > > > > > > > >> I don't know if that is something we would want to
> change, but
> > > > > I'll be
> > > > > > > > >> updating the KIP for now to reflect this. Perhaps we would
> > > want to
> > > > > > > add more
> > > > > > > > >> to the documentation of the the producer configs to
> clarify.
> > > > > > > > >>
> > > > > > > > >> Thank you,
> > > > > > > > >> Justine
> > > > > > > > >>
> > > > > > > > >> On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > > > > jolshan@confluent.io>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi Colin,
> > > > > > > > >>>
> > > > > > > > >>> Thanks for looking at the KIP. I can definitely add to
> the
> > > title
> > > > > to
> > > > > > > make
> > > > > > > > >>> it more clear.
> > > > > > > > >>>
> > > > > > > > >>> It makes sense that both configurations could be turned
> on
> > > since
> > > > > > > there
> > > > > > > > >>> are many cases where the user can not control the
> server-side
> > > > > > > > >>> configurations. I was a little concerned about how both
> > > > > interacting
> > > > > > > would
> > > > > > > > >>> work out -- if there would be to many requests for new
> > > topics,
> > > > > for
> > > > > > > example.
> > > > > > > > >>> But it since it does make sense to allow both
> configurations
> > > > > > > enabled, I
> > > > > > > > >>> will test out some scenarios and I'll change the KIP to
> > > support
> > > > > this.
> > > > > > > > >>>
> > > > > > > > >>> I also agree with documentation about distinguishing the
> > > > > > > differences. I
> > > > > > > > >>> was having some trouble with the wording but I like the
> > > phrases
> > > > > > > > >>> "server-side" and "client-side." That's a good
> distinction I
> > > can
> > > > > use
> > > > > > > when
> > > > > > > > >>> describing.
> > > > > > > > >>>
> > > > > > > > >>> I'll try to update the KIP soon keeping everyone's input
> in
> > > mind.
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>> Justine
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > > cmccabe@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hi Justine,
> > > > > > > > >>>>
> > > > > > > > >>>> Thanks for the KIP.  This seems like a good step towards
> > > > > removing
> > > > > > > > >>>> server-side topic auto-creation.
> > > > > > > > >>>>
> > > > > > > > >>>> We should add included "client-side" to the title of
> the KIP
> > > > > > > somewhere,
> > > > > > > > >>>> to make it clear that we're talking about client-side
> auto
> > > > > creation.
> > > > > > > > >>>>
> > > > > > > > >>>> The KIP says:
> > > > > > > > >>>> > In order to automatically create topics with the
> > > producer, the
> > > > > > > > >>>> producer's
> > > > > > > > >>>> > auto.create.topics.enable config must be set to true
> and
> > > the
> > > > > > > broker
> > > > > > > > >>>> config should be set to false
> > > > > > > > >>>>
> > > > > > > > >>>> From a user's point of view, this seems
> counter-intuitive.
> > > In
> > > > > > > order to
> > > > > > > > >>>> auto-create topics the broker's
> auto.create.topics.enable
> > > config
> > > > > > > should be
> > > > > > > > >>>> set to false?  It seems like the server-side
> auto-create is
> > > > > > > unrelated to
> > > > > > > > >>>> the client-side auto-create.  We could have both turned
> on
> > > (and
> > > > > I'm
> > > > > > > sure
> > > > > > > > >>>> that in the real world, people will try this
> > > configuration...)
> > > > > > > There's no
> > > > > > > > >>>> reason not to support this, I think.
> > > > > > > > >>>>
> > > > > > > > >>>> We should add some documentation explaining the
> difference
> > > > > between
> > > > > > > > >>>> server-side and client-side auto-creation.  Without
> > > > > documentation,
> > > > > > > an admin
> > > > > > > > >>>> might think that they had disabled all forms of
> > > auto-creation by
> > > > > > > setting
> > > > > > > > >>>> the -side setting to false-- but this is not the case,
> of
> > > > > course.
> > > > > > > > >>>>
> > > > > > > > >>>> best,
> > > > > > > > >>>> Colin
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > > > > > > >>>> > Hi Dhruvil,
> > > > > > > > >>>> >
> > > > > > > > >>>> > Thanks for reading the KIP!
> > > > > > > > >>>> > That was the general idea for deprecation. We would
> log a
> > > > > warning
> > > > > > > > >>>> when the
> > > > > > > > >>>> > config is enabled on the broker.
> > > > > > > > >>>> > I also believe that there would be a change to
> > > documentation.
> > > > > > > > >>>> > If there is anything else that should be done, please
> let
> > > me
> > > > > know!
> > > > > > > > >>>> >
> > > > > > > > >>>> > Justine
> > > > > > > > >>>> >
> > > > > > > > >>>> > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > > > > > > dhruvil@confluent.io>
> > > > > > > > >>>> wrote:
> > > > > > > > >>>> >
> > > > > > > > >>>> > > Hi Justine,
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > Thanks for the KIP, this is great!
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > Could you add some more information about what
> > > deprecating
> > > > > the
> > > > > > > > >>>> broker
> > > > > > > > >>>> > > configuration means? Would we log a warning in the
> logs
> > > when
> > > > > > > auto
> > > > > > > > >>>> topic
> > > > > > > > >>>> > > creation is enabled on the broker, for example?
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > Thanks,
> > > > > > > > >>>> > > Dhruvil
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > > > > > > > >>>> jolshan@confluent.io>
> > > > > > > > >>>> > > wrote:
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > > Hello all,
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > I'd like to start a discussion thread for KIP-487.
> > > > > > > > >>>> > > > This KIP plans to deprecate the current system of
> > > > > > > auto-creating
> > > > > > > > >>>> topics
> > > > > > > > >>>> > > > through requests to the metadata and give the
> > > producer the
> > > > > > > > >>>> ability to
> > > > > > > > >>>> > > > automatically create topics instead.
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > More information can be found here:
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > >
> > > > > > > > >>>>
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > Thank you,
> > > > > > > > >>>> > > > Justine Olshan
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > >
> > > > > > > > >>>> >
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > >
> > > > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Satish Duggana <sa...@gmail.com>.

Hi Justine,
Thanks for the clarifications.

I understand that auto-creation of topics will happen through
CreateTopic request instead of metadata request. What I meant in
earlier mail is producer client should not override broker config
about auto-creation of topics. I agree with Harsha on other mail about
this behavior. If auto-creation is disabled on broker, producer
clients will never be allowed to create topics even if
'allow.auto.create.topics' is true in producer client.

On Wed, Aug 7, 2019 at 1:28 AM Justine Olshan <jo...@confluent.io> wrote:
>
> Hi Satish,
>
> Thanks for looking at the KIP.
>
> Yes, the producer will wait for the topic to be created before it can send
> any messages to it.
>
> I would like to clarify "overriding" broker behavior. If the client enables
> client-side autocreation, the only difference will be that the topic
> auto-creation will no longer occur in the metadata request and will instead
> come from a CreateTopic request on the producer.
> Partitions and replication factor will be determined by the broker configs.
>
> Is this similar to what you were thinking? Please let me know if there is
> something you think I missed.
>
> Thank you,
> Justine
>
> On Tue, Aug 6, 2019 at 12:01 PM Satish Duggana <sa...@gmail.com>
> wrote:
>
> > Hi Justine,
> > Thanks for the KIP. This is a nice addition to the producer client
> > without running admin-client’s create topic APIs. Does producer wait
> > for the topic to be created successfully before it tries to publish
> > messages to that topic? I assume that this will not throw an error
> > that the topic does not exist.
> >
> > As mentioned by others, overriding broker behavior by producer looks
> > to be a concern. IMHO, broker should have a way to use either default
> > constraints or configure custom constraints before these can be
> > overridden by clients but not vice versa. There should be an option on
> > brokers whether those constraints can be overridden by producers or
> > not.
> >
> > Thanks,
> > Satish.
> >
> > On Tue, Aug 6, 2019 at 11:39 PM Justine Olshan <jo...@confluent.io>
> > wrote:
> > >
> > > Hi Harsha,
> > >
> > > After taking this all into consideration, I've updated the KIP to no
> > longer
> > > allow client-side configuration of replication factor and partitions.
> > > Instead, the broker defaults will be used as long as the broker supports
> > > KIP 464.
> > > If the broker does not support this KIP, then the client can not create
> > > topics on its own. (Behavior that exists now)
> > >
> > > I think this will help with your concerns. Please let me know if you
> > > further feedback.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Tue, Aug 6, 2019 at 10:49 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> > >
> > > > Hi,
> > > >     Even with policies one needs to implement that, so for every user
> > who
> > > > doesn't want a producer to create topics or have limits around
> > partitions &
> > > > replication factor they have to implement a policy.
> > > >       The KIP is changing the behavior , it might not be introducing
> > the
> > > > new functionality but it will enable producers to override the create
> > topic
> > > > config settings on the broker. What I am asking for to provide a config
> > > > that will disable auto creation of topics and if its enabled set some
> > sane
> > > > defaults so that clients can create a topic with in those limits. I
> > don't
> > > > see how this not related to this KIP.
> > > >      If the server config options as I suggested doesn't interest you
> > at
> > > > least have a default CreateTopicPolices in place.
> > > >        To give an example, In our environment we disable the
> > > > auto.create.topic.enable and force users to go through a centralized
> > > > service as we want capture more details about the topic creation and
> > > > requirements. With this KIP, a producer can create a topic with no
> > bounds.
> > > >  Another example max.message.size we define that at cluster level and
> > one
> > > > can override max.messsage.size at topic level, any misconfiguration at
> > this
> > > > will cause service degradation.  Its not always about the rogue
> > clients,
> > > > users can easily misconfigure and can cause an outage.
> > > > Again we can talk about CreateTopicPolicy but without having a default
> > > > implementation and asking everyone to implement their own while
> > changing
> > > > the behavior in producer  doesn't make sense to me.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Hi Harsha,
> > > > >
> > > > > I mentioned policies and the authorizer. For example, with
> > > > > CreateTopicPolicy, you can implement the limits you describe. If you
> > have
> > > > > ideas of how that should be improved, please submit a KIP. My point
> > is
> > > > that
> > > > > this KIP is not introducing any new functionality with regards to
> > what
> > > > > rogue clients can do. It's using the existing protocol that is
> > already
> > > > > exposed via the AdminClient. So, I don't think we need to address it
> > in
> > > > > this KIP. Does that make sense?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io>
> > > > > wrote:
> > > > >
> > > > > Ismael,
> > > > > Sure AdminClient can do that and we should've shipped a config or
> > use the
> > > > > existing one to block that. Not all users are yet to upgrade to
> > > > AdminClient
> > > > > and start using that to cause issues yet. In shared environment we
> > should
> > > > > allow server to set sane defaults and not allow every client to go
> > ahead
> > > > > create random no.of topic/partitions and replication factor. Even if
> > the
> > > > > users want to allow topic creation proposed in the KIP , it makes
> > sense
> > > > to
> > > > > have some guards against the no.of partitions and replication factor.
> > > > > Authorizer is not always an answer to block requests and having
> > users set
> > > > > server side configs to protect a multi-tenant environment is
> > required.
> > > > In a
> > > > > non-secure environment Authorizer is a blunt instrument either you
> > end up
> > > > > blocking everyone or allowing everyone.
> > > > > I am asking to have server side that allow clients to create topics
> > or
> > > > not
> > > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > > replication-factor.
> > > > >
> > > > > -Harsha
> > > > >
> > > > > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> > > > >
> > > > > Harsha,
> > > > >
> > > > > Rogue clients can use the admin client to create topics and
> > partitions.
> > > > > ACLs and policies can help in that case as well as this one.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > > Thanks for the KIP.
> > > > > "When server-side auto-creation is disabled, client-side
> > auto-creation
> > > > > will try to use client-side configurations"
> > > > > If I understand correctly, this KIP is removing any server-side
> > blocking
> > > > > client auto creation of topic?
> > > > > if so this will present potential issue of rogue client creating ton
> > of
> > > > > topic-partitions and potentially bringing down the service for
> > everyone
> > > > >
> > > > > or
> > > > >
> > > > > degrade the service itself.
> > > > > By reading the KIP its not clear to me that there is a clear way to
> > block
> > > > > auto creation topics of all together from clients by server side
> > config.
> > > > > Server side configs of default topic, partitions should take higher
> > > > > precedence and client shouldn't be able to create a topic with higher
> > > > >
> > > > > no.of
> > > > >
> > > > > partitions, replication than what server config specifies.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <
> > jolshan@confluent.io>
> > > > > wrote:
> > > > >
> > > > > Hi all,
> > > > > I made some changes to the KIP. Hopefully this configuration change
> > > > >
> > > > > will
> > > > >
> > > > > make things a little clearer.
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > > >
> > > > > Please let me know if you have any feedback or questions!
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I think you bring up a good point. It would be better if we didn't
> > ever
> > > > > have to set up client-side configuration for this feature, and
> > KIP-464
> > > > > would let us skip this entirely.
> > > > >
> > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > > >
> > > > > Hi Mickael,
> > > > > I agree that KIP-464 works on newer brokers, but I was a bit worried
> > > > >
> > > > > how
> > > > >
> > > > > things would play out on older brokers that* do not *have KIP 464
> > > > >
> > > > > included.
> > > > >
> > > > > Is it enough to throw an error in this case when producer configs are
> > > > >
> > > > > not
> > > > >
> > > > > specified?
> > > > >
> > > > > I think the right thing to do would be to log an error message in the
> > > > > client. We will need to have that capability in any case, to cover
> > > > > scenarios like the client trying to auto-create a topic that they
> > don't
> > > > > have permission to create. Or a client trying to create a topic on a
> > > > >
> > > > > broker
> > > > >
> > > > > so old that CreateTopicsRequest is not supported.
> > > > >
> > > > > The big downside to relying on KIP-464 is that it is a very recent
> > > > >
> > > > > feature
> > > > >
> > > > > -- so recent that it hasn't even made its way to any official Apache
> > > > > release. It's scheduled for the upcoming 2.4 release in a few months.
> > > > >
> > > > > So if you view this KIP as a step towards removing broker-side
> > > > > auto-create, you might want to support older brokers just to
> > accelerate
> > > > > adoption, and hasten the day when we can finally flip broker-side
> > > > > auto-create to off (or even remove it entirely).
> > > > >
> > > > > I have to agree, though, that having client-side configurations for
> > > > >
> > > > > number
> > > > >
> > > > > of partitions and replication factor is messy. Maybe it would be
> > worth
> > > > >
> > > > > it
> > > > >
> > > > > to restrict support to post-KIP-464 brokers, if we could avoid
> > creating
> > > > > more configs.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> > > > >
> > > > > mickael.maison@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > We can rely on KIP-464 which allows to omit the partition count or
> > > > > replication factor when creating a topic. In that case, the broker
> > > > >
> > > > > defaults
> > > > >
> > > > > are used.
> > > > >
> > > > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jolshan@confluent.io
> > >
> > > > > wrote:
> > > > >
> > > > > Michael,
> > > > > That makes sense to me!
> > > > > To clarify, in the current state of the KIP, the producer does not
> > > > >
> > > > > rely
> > > > >
> > > > > on
> > > > >
> > > > > the broker to autocreate--if the broker's config is disabled, then
> > > > >
> > > > > the
> > > > >
> > > > > producer can autocreate on its own with a create topic request (the
> > > > >
> > > > > same
> > > > >
> > > > > type of request the admin client uses).
> > > > > However, if both configs are enabled, the broker will autocreate
> > > > >
> > > > > through
> > > > >
> > > > > a
> > > > >
> > > > > metadata request before the producer gets a chance. Of course, the
> > way
> > > > >
> > > > > to
> > > > >
> > > > > avoid this, is to do as you suggested, and set
> > > > >
> > > > > the
> > > > >
> > > > > "allow_auto_topic_creation" field to false.
> > > > >
> > > > > I think the only thing we need to be careful with in this setup is
> > > > >
> > > > > without
> > > > >
> > > > > KIP 464, we can not use broker defaults for this topic. A user needs
> > > > >
> > > > > to
> > > > >
> > > > > specify the number of partition and replication factor in the config.
> > > > >
> > > > > An
> > > > >
> > > > > alternative to this is to have coded defaults for when these
> > > > >
> > > > > configs
> > > > >
> > > > > are
> > > > >
> > > > > unspecified, but it is not immediately apparent what these defaults
> > > > >
> > > > > should
> > > > >
> > > > > be.
> > > > >
> > > > > Thanks again for reading my KIP,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > > > >
> > > > > mickael.maison@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the response!
> > > > > In my opinion, it would be better if the producer did not rely at
> > > > >
> > > > > all
> > > > >
> > > > > on the broker auto create feature as this is what we're aiming to
> > > > > deprecate. When requesting metadata we can set the
> > > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > > creation. Then if the topic is not existing, send a
> > CreateTopicRequest.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Currently the way it is implemented, the broker auto-creation
> > > > >
> > > > > configuration
> > > > >
> > > > > takes precedence. The producer will not use the CreateTopics
> > > > >
> > > > > request.
> > > > >
> > > > > (Technically it can--but the topic will already be created
> > > > >
> > > > > through
> > > > >
> > > > > the
> > > > >
> > > > > broker, so it will never try to create the topic.) It is possible to
> > > > > change this however, and I'd be happy to
> > > > >
> > > > > discuss
> > > > >
> > > > > the
> > > > >
> > > > > benefits of this alternative.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > > > >
> > > > > mickael.maison@gmail.com>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP!
> > > > >
> > > > > In case auto creation is enabled on both the client and server,
> > > > >
> > > > > will
> > > > >
> > > > > the producer still use the AdminClient (CreateTopics request)
> > > > >
> > > > > to
> > > > >
> > > > > create topics? and not rely on the broker auto create. I'm guessing
> > the
> > > > > answer is yes but can you make it explicit.
> > > > >
> > > > > Thank you
> > > > >
> > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi,
> > > > > Just a friendly reminder to take a look at this KIP if you
> > > > >
> > > > > have
> > > > >
> > > > > the
> > > > >
> > > > > time.
> > > > >
> > > > > I was thinking about broker vs. client default precedence,
> > > > >
> > > > > and I
> > > > >
> > > > > think it
> > > > >
> > > > > makes sense to keep the broker as the default used when both
> > > > >
> > > > > client-side
> > > > >
> > > > > and broker-side defaults are configured. The idea is that
> > > > >
> > > > > there
> > > > >
> > > > > would be
> > > > >
> > > > > pretty clear documentation, and that many systems with
> > > > >
> > > > > configurations
> > > > >
> > > > > that
> > > > >
> > > > > the client could not change would likely have the auto-create
> > > > >
> > > > > default
> > > > >
> > > > > off.
> > > > >
> > > > > (In cloud for example).
> > > > >
> > > > > It also seems like in most cases, the consumer config
> > > > > 'allow.auto.create.topics' was created to actually prevent
> > > > >
> > > > > the
> > > > >
> > > > > creation
> > > > >
> > > > > of
> > > > >
> > > > > topics, so the loss of creation functionality will not be a
> > > > >
> > > > > big
> > > > >
> > > > > problem.
> > > > >
> > > > > I'm happy to discuss any other compatibility problems or
> > > > >
> > > > > components
> > > > >
> > > > > of
> > > > >
> > > > > this KIP.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > I was looking at this KIP again, and there is a decision I
> > > > >
> > > > > made
> > > > >
> > > > > that I
> > > > >
> > > > > think is worth discussing.
> > > > >
> > > > > In the case where both the broker and producer's
> > > > > 'auto.create.topics.enable' are set to true, we have to
> > > > >
> > > > > choose
> > > > >
> > > > > either
> > > > >
> > > > > the
> > > > >
> > > > > broker configs or the producer configs for the replication
> > > > > factor/partitions.
> > > > >
> > > > > Currently, the decision is to have the broker defaults take
> > > > >
> > > > > precedence.
> > > > >
> > > > > (It is easier to do this in the implementation.) It also
> > > > >
> > > > > makes
> > > > >
> > > > > some
> > > > >
> > > > > sense
> > > > >
> > > > > for this behavior to take precedence since this behavior
> > > > >
> > > > > already
> > > > >
> > > > > occurs as
> > > > >
> > > > > the default.
> > > > >
> > > > > However, I was wondering if it would be odd for those who
> > > > >
> > > > > can
> > > > >
> > > > > only
> > > > >
> > > > > see
> > > > >
> > > > > the
> > > > >
> > > > > producer side to set configs for replication
> > > > >
> > > > > factor/partitions
> > > > >
> > > > > and
> > > > >
> > > > > see
> > > > >
> > > > > different results. Currently the documentation for the
> > > > >
> > > > > config
> > > > >
> > > > > states
> > > > >
> > > > > that
> > > > >
> > > > > the config values are only used when the broker config is
> > > > >
> > > > > not
> > > > >
> > > > > enabled,
> > > > >
> > > > > but
> > > > >
> > > > > this might not always be clear to the user. Changing the
> > > > >
> > > > > code
> > > > >
> > > > > to
> > > > >
> > > > > have
> > > > >
> > > > > the
> > > > >
> > > > > producer's configurations take precedence is possible, but
> > > > >
> > > > > I
> > > > >
> > > > > was
> > > > >
> > > > > wondering
> > > > >
> > > > > what everyone thought.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Just a quick update--
> > > > >
> > > > > It seems that enabling both the broker and producer
> > > > >
> > > > > configs
> > > > >
> > > > > works
> > > > >
> > > > > fine,
> > > > >
> > > > > except that the broker configurations for partitions,
> > > > >
> > > > > replication
> > > > >
> > > > > factor
> > > > >
> > > > > take precedence.
> > > > > I don't know if that is something we would want to
> > > > >
> > > > > change, but
> > > > >
> > > > > I'll be
> > > > >
> > > > > updating the KIP for now to reflect this. Perhaps we would
> > > > >
> > > > > want to
> > > > >
> > > > > add more
> > > > >
> > > > > to the documentation of the the producer configs to
> > > > >
> > > > > clarify.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Colin,
> > > > >
> > > > > Thanks for looking at the KIP. I can definitely add to
> > > > >
> > > > > the
> > > > >
> > > > > title
> > > > >
> > > > > to
> > > > >
> > > > > make
> > > > >
> > > > > it more clear.
> > > > >
> > > > > It makes sense that both configurations could be turned
> > > > >
> > > > > on
> > > > >
> > > > > since
> > > > >
> > > > > there
> > > > >
> > > > > are many cases where the user can not control the
> > > > >
> > > > > server-side
> > > > >
> > > > > configurations. I was a little concerned about how both
> > > > >
> > > > > interacting
> > > > >
> > > > > would
> > > > >
> > > > > work out -- if there would be to many requests for new
> > > > >
> > > > > topics,
> > > > >
> > > > > for
> > > > >
> > > > > example.
> > > > >
> > > > > But it since it does make sense to allow both
> > > > >
> > > > > configurations
> > > > >
> > > > > enabled, I
> > > > >
> > > > > will test out some scenarios and I'll change the KIP to
> > > > >
> > > > > support
> > > > >
> > > > > this.
> > > > >
> > > > > I also agree with documentation about distinguishing the
> > > > >
> > > > > differences. I
> > > > >
> > > > > was having some trouble with the wording but I like the
> > > > >
> > > > > phrases
> > > > >
> > > > > "server-side" and "client-side." That's a good
> > > > >
> > > > > distinction I
> > > > >
> > > > > can
> > > > >
> > > > > use
> > > > >
> > > > > when
> > > > >
> > > > > describing.
> > > > >
> > > > > I'll try to update the KIP soon keeping everyone's input
> > > > >
> > > > > in
> > > > >
> > > > > mind.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > > > >
> > > > > cmccabe@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP. This seems like a good step towards
> > > > >
> > > > > removing
> > > > >
> > > > > server-side topic auto-creation.
> > > > >
> > > > > We should add included "client-side" to the title of
> > > > >
> > > > > the KIP
> > > > >
> > > > > somewhere,
> > > > >
> > > > > to make it clear that we're talking about client-side
> > > > >
> > > > > auto
> > > > >
> > > > > creation.
> > > > >
> > > > > The KIP says:
> > > > >
> > > > > In order to automatically create topics with the
> > > > >
> > > > > producer, the
> > > > >
> > > > > producer's
> > > > >
> > > > > auto.create.topics.enable config must be set to true
> > > > >
> > > > > and
> > > > >
> > > > > the
> > > > >
> > > > > broker
> > > > >
> > > > > config should be set to false
> > > > >
> > > > > From a user's point of view, this seems
> > > > >
> > > > > counter-intuitive.
> > > > >
> > > > > In
> > > > >
> > > > > order to
> > > > >
> > > > > auto-create topics the broker's
> > > > >
> > > > > auto.create.topics.enable
> > > > >
> > > > > config
> > > > >
> > > > > should be
> > > > >
> > > > > set to false? It seems like the server-side
> > > > >
> > > > > auto-create is
> > > > >
> > > > > unrelated to
> > > > >
> > > > > the client-side auto-create. We could have both turned
> > > > >
> > > > > on
> > > > >
> > > > > (and
> > > > >
> > > > > I'm
> > > > >
> > > > > sure
> > > > >
> > > > > that in the real world, people will try this
> > > > >
> > > > > configuration...)
> > > > >
> > > > > There's no
> > > > >
> > > > > reason not to support this, I think.
> > > > >
> > > > > We should add some documentation explaining the
> > > > >
> > > > > difference
> > > > >
> > > > > between
> > > > >
> > > > > server-side and client-side auto-creation. Without
> > > > >
> > > > > documentation,
> > > > >
> > > > > an admin
> > > > >
> > > > > might think that they had disabled all forms of
> > > > >
> > > > > auto-creation by
> > > > >
> > > > > setting
> > > > >
> > > > > the -side setting to false-- but this is not the case,
> > > > >
> > > > > of
> > > > >
> > > > > course.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > > >
> > > > > Hi Dhruvil,
> > > > >
> > > > > Thanks for reading the KIP!
> > > > > That was the general idea for deprecation. We would
> > > > >
> > > > > log a
> > > > >
> > > > > warning
> > > > >
> > > > > when the
> > > > >
> > > > > config is enabled on the broker.
> > > > > I also believe that there would be a change to
> > > > >
> > > > > documentation.
> > > > >
> > > > > If there is anything else that should be done, please
> > > > >
> > > > > let
> > > > >
> > > > > me
> > > > >
> > > > > know!
> > > > >
> > > > > Justine
> > > > >
> > > > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > > > >
> > > > > dhruvil@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP, this is great!
> > > > >
> > > > > Could you add some more information about what
> > > > >
> > > > > deprecating
> > > > >
> > > > > the
> > > > >
> > > > > broker
> > > > >
> > > > > configuration means? Would we log a warning in the
> > > > >
> > > > > logs
> > > > >
> > > > > when
> > > > >
> > > > > auto
> > > > >
> > > > > topic
> > > > >
> > > > > creation is enabled on the broker, for example?
> > > > >
> > > > > Thanks,
> > > > > Dhruvil
> > > > >
> > > > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > > > > deprecate the current system of
> > > > >
> > > > > auto-creating
> > > > >
> > > > > topics
> > > > >
> > > > > through requests to the metadata and give the
> > > > >
> > > > > producer the
> > > > >
> > > > > ability to
> > > > >
> > > > > automatically create topics instead.
> > > > >
> > > > > More information can be found here:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > > >
> > > > > Thank you,
> > > > > Justine Olshan
> > > > >
> > > > >
> > > >
> >

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Justine Olshan <jo...@confluent.io>.

Hi Satish,

Thanks for looking at the KIP.

Yes, the producer will wait for the topic to be created before it can send
any messages to it.

I would like to clarify "overriding" broker behavior. If the client enables
client-side autocreation, the only difference will be that the topic
auto-creation will no longer occur in the metadata request and will instead
come from a CreateTopic request on the producer.
Partitions and replication factor will be determined by the broker configs.

Is this similar to what you were thinking? Please let me know if there is
something you think I missed.

Thank you,
Justine

On Tue, Aug 6, 2019 at 12:01 PM Satish Duggana <sa...@gmail.com>
wrote:

> Hi Justine,
> Thanks for the KIP. This is a nice addition to the producer client
> without running admin-client’s create topic APIs. Does producer wait
> for the topic to be created successfully before it tries to publish
> messages to that topic? I assume that this will not throw an error
> that the topic does not exist.
>
> As mentioned by others, overriding broker behavior by producer looks
> to be a concern. IMHO, broker should have a way to use either default
> constraints or configure custom constraints before these can be
> overridden by clients but not vice versa. There should be an option on
> brokers whether those constraints can be overridden by producers or
> not.
>
> Thanks,
> Satish.
>
> On Tue, Aug 6, 2019 at 11:39 PM Justine Olshan <jo...@confluent.io>
> wrote:
> >
> > Hi Harsha,
> >
> > After taking this all into consideration, I've updated the KIP to no
> longer
> > allow client-side configuration of replication factor and partitions.
> > Instead, the broker defaults will be used as long as the broker supports
> > KIP 464.
> > If the broker does not support this KIP, then the client can not create
> > topics on its own. (Behavior that exists now)
> >
> > I think this will help with your concerns. Please let me know if you
> > further feedback.
> >
> > Thank you,
> > Justine
> >
> > On Tue, Aug 6, 2019 at 10:49 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
> >
> > > Hi,
> > >     Even with policies one needs to implement that, so for every user
> who
> > > doesn't want a producer to create topics or have limits around
> partitions &
> > > replication factor they have to implement a policy.
> > >       The KIP is changing the behavior , it might not be introducing
> the
> > > new functionality but it will enable producers to override the create
> topic
> > > config settings on the broker. What I am asking for to provide a config
> > > that will disable auto creation of topics and if its enabled set some
> sane
> > > defaults so that clients can create a topic with in those limits. I
> don't
> > > see how this not related to this KIP.
> > >      If the server config options as I suggested doesn't interest you
> at
> > > least have a default CreateTopicPolices in place.
> > >        To give an example, In our environment we disable the
> > > auto.create.topic.enable and force users to go through a centralized
> > > service as we want capture more details about the topic creation and
> > > requirements. With this KIP, a producer can create a topic with no
> bounds.
> > >  Another example max.message.size we define that at cluster level and
> one
> > > can override max.messsage.size at topic level, any misconfiguration at
> this
> > > will cause service degradation.  Its not always about the rogue
> clients,
> > > users can easily misconfigure and can cause an outage.
> > > Again we can talk about CreateTopicPolicy but without having a default
> > > implementation and asking everyone to implement their own while
> changing
> > > the behavior in producer  doesn't make sense to me.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > Hi Harsha,
> > > >
> > > > I mentioned policies and the authorizer. For example, with
> > > > CreateTopicPolicy, you can implement the limits you describe. If you
> have
> > > > ideas of how that should be improved, please submit a KIP. My point
> is
> > > that
> > > > this KIP is not introducing any new functionality with regards to
> what
> > > > rogue clients can do. It's using the existing protocol that is
> already
> > > > exposed via the AdminClient. So, I don't think we need to address it
> in
> > > > this KIP. Does that make sense?
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io>
> > > > wrote:
> > > >
> > > > Ismael,
> > > > Sure AdminClient can do that and we should've shipped a config or
> use the
> > > > existing one to block that. Not all users are yet to upgrade to
> > > AdminClient
> > > > and start using that to cause issues yet. In shared environment we
> should
> > > > allow server to set sane defaults and not allow every client to go
> ahead
> > > > create random no.of topic/partitions and replication factor. Even if
> the
> > > > users want to allow topic creation proposed in the KIP , it makes
> sense
> > > to
> > > > have some guards against the no.of partitions and replication factor.
> > > > Authorizer is not always an answer to block requests and having
> users set
> > > > server side configs to protect a multi-tenant environment is
> required.
> > > In a
> > > > non-secure environment Authorizer is a blunt instrument either you
> end up
> > > > blocking everyone or allowing everyone.
> > > > I am asking to have server side that allow clients to create topics
> or
> > > not
> > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > replication-factor.
> > > >
> > > > -Harsha
> > > >
> > > > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> > > >
> > > > Harsha,
> > > >
> > > > Rogue clients can use the admin client to create topics and
> partitions.
> > > > ACLs and policies can help in that case as well as this one.
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > > Thanks for the KIP.
> > > > "When server-side auto-creation is disabled, client-side
> auto-creation
> > > > will try to use client-side configurations"
> > > > If I understand correctly, this KIP is removing any server-side
> blocking
> > > > client auto creation of topic?
> > > > if so this will present potential issue of rogue client creating ton
> of
> > > > topic-partitions and potentially bringing down the service for
> everyone
> > > >
> > > > or
> > > >
> > > > degrade the service itself.
> > > > By reading the KIP its not clear to me that there is a clear way to
> block
> > > > auto creation topics of all together from clients by server side
> config.
> > > > Server side configs of default topic, partitions should take higher
> > > > precedence and client shouldn't be able to create a topic with higher
> > > >
> > > > no.of
> > > >
> > > > partitions, replication than what server config specifies.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <
> jolshan@confluent.io>
> > > > wrote:
> > > >
> > > > Hi all,
> > > > I made some changes to the KIP. Hopefully this configuration change
> > > >
> > > > will
> > > >
> > > > make things a little clearer.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > >
> > > > Please let me know if you have any feedback or questions!
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> > > >
> > > > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think you bring up a good point. It would be better if we didn't
> ever
> > > > have to set up client-side configuration for this feature, and
> KIP-464
> > > > would let us skip this entirely.
> > > >
> > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > >
> > > > Hi Mickael,
> > > > I agree that KIP-464 works on newer brokers, but I was a bit worried
> > > >
> > > > how
> > > >
> > > > things would play out on older brokers that* do not *have KIP 464
> > > >
> > > > included.
> > > >
> > > > Is it enough to throw an error in this case when producer configs are
> > > >
> > > > not
> > > >
> > > > specified?
> > > >
> > > > I think the right thing to do would be to log an error message in the
> > > > client. We will need to have that capability in any case, to cover
> > > > scenarios like the client trying to auto-create a topic that they
> don't
> > > > have permission to create. Or a client trying to create a topic on a
> > > >
> > > > broker
> > > >
> > > > so old that CreateTopicsRequest is not supported.
> > > >
> > > > The big downside to relying on KIP-464 is that it is a very recent
> > > >
> > > > feature
> > > >
> > > > -- so recent that it hasn't even made its way to any official Apache
> > > > release. It's scheduled for the upcoming 2.4 release in a few months.
> > > >
> > > > So if you view this KIP as a step towards removing broker-side
> > > > auto-create, you might want to support older brokers just to
> accelerate
> > > > adoption, and hasten the day when we can finally flip broker-side
> > > > auto-create to off (or even remove it entirely).
> > > >
> > > > I have to agree, though, that having client-side configurations for
> > > >
> > > > number
> > > >
> > > > of partitions and replication factor is messy. Maybe it would be
> worth
> > > >
> > > > it
> > > >
> > > > to restrict support to post-KIP-464 brokers, if we could avoid
> creating
> > > > more configs.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> > > >
> > > > mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > We can rely on KIP-464 which allows to omit the partition count or
> > > > replication factor when creating a topic. In that case, the broker
> > > >
> > > > defaults
> > > >
> > > > are used.
> > > >
> > > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jolshan@confluent.io
> >
> > > > wrote:
> > > >
> > > > Michael,
> > > > That makes sense to me!
> > > > To clarify, in the current state of the KIP, the producer does not
> > > >
> > > > rely
> > > >
> > > > on
> > > >
> > > > the broker to autocreate--if the broker's config is disabled, then
> > > >
> > > > the
> > > >
> > > > producer can autocreate on its own with a create topic request (the
> > > >
> > > > same
> > > >
> > > > type of request the admin client uses).
> > > > However, if both configs are enabled, the broker will autocreate
> > > >
> > > > through
> > > >
> > > > a
> > > >
> > > > metadata request before the producer gets a chance. Of course, the
> way
> > > >
> > > > to
> > > >
> > > > avoid this, is to do as you suggested, and set
> > > >
> > > > the
> > > >
> > > > "allow_auto_topic_creation" field to false.
> > > >
> > > > I think the only thing we need to be careful with in this setup is
> > > >
> > > > without
> > > >
> > > > KIP 464, we can not use broker defaults for this topic. A user needs
> > > >
> > > > to
> > > >
> > > > specify the number of partition and replication factor in the config.
> > > >
> > > > An
> > > >
> > > > alternative to this is to have coded defaults for when these
> > > >
> > > > configs
> > > >
> > > > are
> > > >
> > > > unspecified, but it is not immediately apparent what these defaults
> > > >
> > > > should
> > > >
> > > > be.
> > > >
> > > > Thanks again for reading my KIP,
> > > > Justine
> > > >
> > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > > >
> > > > mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the response!
> > > > In my opinion, it would be better if the producer did not rely at
> > > >
> > > > all
> > > >
> > > > on the broker auto create feature as this is what we're aiming to
> > > > deprecate. When requesting metadata we can set the
> > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > creation. Then if the topic is not existing, send a
> CreateTopicRequest.
> > > >
> > > > What do you think?
> > > >
> > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Currently the way it is implemented, the broker auto-creation
> > > >
> > > > configuration
> > > >
> > > > takes precedence. The producer will not use the CreateTopics
> > > >
> > > > request.
> > > >
> > > > (Technically it can--but the topic will already be created
> > > >
> > > > through
> > > >
> > > > the
> > > >
> > > > broker, so it will never try to create the topic.) It is possible to
> > > > change this however, and I'd be happy to
> > > >
> > > > discuss
> > > >
> > > > the
> > > >
> > > > benefits of this alternative.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > > >
> > > > mickael.maison@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > In case auto creation is enabled on both the client and server,
> > > >
> > > > will
> > > >
> > > > the producer still use the AdminClient (CreateTopics request)
> > > >
> > > > to
> > > >
> > > > create topics? and not rely on the broker auto create. I'm guessing
> the
> > > > answer is yes but can you make it explicit.
> > > >
> > > > Thank you
> > > >
> > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Hi,
> > > > Just a friendly reminder to take a look at this KIP if you
> > > >
> > > > have
> > > >
> > > > the
> > > >
> > > > time.
> > > >
> > > > I was thinking about broker vs. client default precedence,
> > > >
> > > > and I
> > > >
> > > > think it
> > > >
> > > > makes sense to keep the broker as the default used when both
> > > >
> > > > client-side
> > > >
> > > > and broker-side defaults are configured. The idea is that
> > > >
> > > > there
> > > >
> > > > would be
> > > >
> > > > pretty clear documentation, and that many systems with
> > > >
> > > > configurations
> > > >
> > > > that
> > > >
> > > > the client could not change would likely have the auto-create
> > > >
> > > > default
> > > >
> > > > off.
> > > >
> > > > (In cloud for example).
> > > >
> > > > It also seems like in most cases, the consumer config
> > > > 'allow.auto.create.topics' was created to actually prevent
> > > >
> > > > the
> > > >
> > > > creation
> > > >
> > > > of
> > > >
> > > > topics, so the loss of creation functionality will not be a
> > > >
> > > > big
> > > >
> > > > problem.
> > > >
> > > > I'm happy to discuss any other compatibility problems or
> > > >
> > > > components
> > > >
> > > > of
> > > >
> > > > this KIP.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > > >
> > > > jolshan@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I was looking at this KIP again, and there is a decision I
> > > >
> > > > made
> > > >
> > > > that I
> > > >
> > > > think is worth discussing.
> > > >
> > > > In the case where both the broker and producer's
> > > > 'auto.create.topics.enable' are set to true, we have to
> > > >
> > > > choose
> > > >
> > > > either
> > > >
> > > > the
> > > >
> > > > broker configs or the producer configs for the replication
> > > > factor/partitions.
> > > >
> > > > Currently, the decision is to have the broker defaults take
> > > >
> > > > precedence.
> > > >
> > > > (It is easier to do this in the implementation.) It also
> > > >
> > > > makes
> > > >
> > > > some
> > > >
> > > > sense
> > > >
> > > > for this behavior to take precedence since this behavior
> > > >
> > > > already
> > > >
> > > > occurs as
> > > >
> > > > the default.
> > > >
> > > > However, I was wondering if it would be odd for those who
> > > >
> > > > can
> > > >
> > > > only
> > > >
> > > > see
> > > >
> > > > the
> > > >
> > > > producer side to set configs for replication
> > > >
> > > > factor/partitions
> > > >
> > > > and
> > > >
> > > > see
> > > >
> > > > different results. Currently the documentation for the
> > > >
> > > > config
> > > >
> > > > states
> > > >
> > > > that
> > > >
> > > > the config values are only used when the broker config is
> > > >
> > > > not
> > > >
> > > > enabled,
> > > >
> > > > but
> > > >
> > > > this might not always be clear to the user. Changing the
> > > >
> > > > code
> > > >
> > > > to
> > > >
> > > > have
> > > >
> > > > the
> > > >
> > > > producer's configurations take precedence is possible, but
> > > >
> > > > I
> > > >
> > > > was
> > > >
> > > > wondering
> > > >
> > > > what everyone thought.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Just a quick update--
> > > >
> > > > It seems that enabling both the broker and producer
> > > >
> > > > configs
> > > >
> > > > works
> > > >
> > > > fine,
> > > >
> > > > except that the broker configurations for partitions,
> > > >
> > > > replication
> > > >
> > > > factor
> > > >
> > > > take precedence.
> > > > I don't know if that is something we would want to
> > > >
> > > > change, but
> > > >
> > > > I'll be
> > > >
> > > > updating the KIP for now to reflect this. Perhaps we would
> > > >
> > > > want to
> > > >
> > > > add more
> > > >
> > > > to the documentation of the the producer configs to
> > > >
> > > > clarify.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Hi Colin,
> > > >
> > > > Thanks for looking at the KIP. I can definitely add to
> > > >
> > > > the
> > > >
> > > > title
> > > >
> > > > to
> > > >
> > > > make
> > > >
> > > > it more clear.
> > > >
> > > > It makes sense that both configurations could be turned
> > > >
> > > > on
> > > >
> > > > since
> > > >
> > > > there
> > > >
> > > > are many cases where the user can not control the
> > > >
> > > > server-side
> > > >
> > > > configurations. I was a little concerned about how both
> > > >
> > > > interacting
> > > >
> > > > would
> > > >
> > > > work out -- if there would be to many requests for new
> > > >
> > > > topics,
> > > >
> > > > for
> > > >
> > > > example.
> > > >
> > > > But it since it does make sense to allow both
> > > >
> > > > configurations
> > > >
> > > > enabled, I
> > > >
> > > > will test out some scenarios and I'll change the KIP to
> > > >
> > > > support
> > > >
> > > > this.
> > > >
> > > > I also agree with documentation about distinguishing the
> > > >
> > > > differences. I
> > > >
> > > > was having some trouble with the wording but I like the
> > > >
> > > > phrases
> > > >
> > > > "server-side" and "client-side." That's a good
> > > >
> > > > distinction I
> > > >
> > > > can
> > > >
> > > > use
> > > >
> > > > when
> > > >
> > > > describing.
> > > >
> > > > I'll try to update the KIP soon keeping everyone's input
> > > >
> > > > in
> > > >
> > > > mind.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > > >
> > > > cmccabe@apache.org
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP. This seems like a good step towards
> > > >
> > > > removing
> > > >
> > > > server-side topic auto-creation.
> > > >
> > > > We should add included "client-side" to the title of
> > > >
> > > > the KIP
> > > >
> > > > somewhere,
> > > >
> > > > to make it clear that we're talking about client-side
> > > >
> > > > auto
> > > >
> > > > creation.
> > > >
> > > > The KIP says:
> > > >
> > > > In order to automatically create topics with the
> > > >
> > > > producer, the
> > > >
> > > > producer's
> > > >
> > > > auto.create.topics.enable config must be set to true
> > > >
> > > > and
> > > >
> > > > the
> > > >
> > > > broker
> > > >
> > > > config should be set to false
> > > >
> > > > From a user's point of view, this seems
> > > >
> > > > counter-intuitive.
> > > >
> > > > In
> > > >
> > > > order to
> > > >
> > > > auto-create topics the broker's
> > > >
> > > > auto.create.topics.enable
> > > >
> > > > config
> > > >
> > > > should be
> > > >
> > > > set to false? It seems like the server-side
> > > >
> > > > auto-create is
> > > >
> > > > unrelated to
> > > >
> > > > the client-side auto-create. We could have both turned
> > > >
> > > > on
> > > >
> > > > (and
> > > >
> > > > I'm
> > > >
> > > > sure
> > > >
> > > > that in the real world, people will try this
> > > >
> > > > configuration...)
> > > >
> > > > There's no
> > > >
> > > > reason not to support this, I think.
> > > >
> > > > We should add some documentation explaining the
> > > >
> > > > difference
> > > >
> > > > between
> > > >
> > > > server-side and client-side auto-creation. Without
> > > >
> > > > documentation,
> > > >
> > > > an admin
> > > >
> > > > might think that they had disabled all forms of
> > > >
> > > > auto-creation by
> > > >
> > > > setting
> > > >
> > > > the -side setting to false-- but this is not the case,
> > > >
> > > > of
> > > >
> > > > course.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > >
> > > > Hi Dhruvil,
> > > >
> > > > Thanks for reading the KIP!
> > > > That was the general idea for deprecation. We would
> > > >
> > > > log a
> > > >
> > > > warning
> > > >
> > > > when the
> > > >
> > > > config is enabled on the broker.
> > > > I also believe that there would be a change to
> > > >
> > > > documentation.
> > > >
> > > > If there is anything else that should be done, please
> > > >
> > > > let
> > > >
> > > > me
> > > >
> > > > know!
> > > >
> > > > Justine
> > > >
> > > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > > >
> > > > dhruvil@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP, this is great!
> > > >
> > > > Could you add some more information about what
> > > >
> > > > deprecating
> > > >
> > > > the
> > > >
> > > > broker
> > > >
> > > > configuration means? Would we log a warning in the
> > > >
> > > > logs
> > > >
> > > > when
> > > >
> > > > auto
> > > >
> > > > topic
> > > >
> > > > creation is enabled on the broker, for example?
> > > >
> > > > Thanks,
> > > > Dhruvil
> > > >
> > > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > > > deprecate the current system of
> > > >
> > > > auto-creating
> > > >
> > > > topics
> > > >
> > > > through requests to the metadata and give the
> > > >
> > > > producer the
> > > >
> > > > ability to
> > > >
> > > > automatically create topics instead.
> > > >
> > > > More information can be found here:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > >
> > > > Thank you,
> > > > Justine Olshan
> > > >
> > > >
> > >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Satish Duggana <sa...@gmail.com>.

Hi Justine,
Thanks for the KIP. This is a nice addition to the producer client
without running admin-client’s create topic APIs. Does producer wait
for the topic to be created successfully before it tries to publish
messages to that topic? I assume that this will not throw an error
that the topic does not exist.

As mentioned by others, overriding broker behavior by producer looks
to be a concern. IMHO, broker should have a way to use either default
constraints or configure custom constraints before these can be
overridden by clients but not vice versa. There should be an option on
brokers whether those constraints can be overridden by producers or
not.

Thanks,
Satish.

On Tue, Aug 6, 2019 at 11:39 PM Justine Olshan <jo...@confluent.io> wrote:
>
> Hi Harsha,
>
> After taking this all into consideration, I've updated the KIP to no longer
> allow client-side configuration of replication factor and partitions.
> Instead, the broker defaults will be used as long as the broker supports
> KIP 464.
> If the broker does not support this KIP, then the client can not create
> topics on its own. (Behavior that exists now)
>
> I think this will help with your concerns. Please let me know if you
> further feedback.
>
> Thank you,
> Justine
>
> On Tue, Aug 6, 2019 at 10:49 AM Harsha Chintalapani <ka...@harsha.io> wrote:
>
> > Hi,
> >     Even with policies one needs to implement that, so for every user who
> > doesn't want a producer to create topics or have limits around partitions &
> > replication factor they have to implement a policy.
> >       The KIP is changing the behavior , it might not be introducing the
> > new functionality but it will enable producers to override the create topic
> > config settings on the broker. What I am asking for to provide a config
> > that will disable auto creation of topics and if its enabled set some sane
> > defaults so that clients can create a topic with in those limits. I don't
> > see how this not related to this KIP.
> >      If the server config options as I suggested doesn't interest you at
> > least have a default CreateTopicPolices in place.
> >        To give an example, In our environment we disable the
> > auto.create.topic.enable and force users to go through a centralized
> > service as we want capture more details about the topic creation and
> > requirements. With this KIP, a producer can create a topic with no bounds.
> >  Another example max.message.size we define that at cluster level and one
> > can override max.messsage.size at topic level, any misconfiguration at this
> > will cause service degradation.  Its not always about the rogue clients,
> > users can easily misconfigure and can cause an outage.
> > Again we can talk about CreateTopicPolicy but without having a default
> > implementation and asking everyone to implement their own while changing
> > the behavior in producer  doesn't make sense to me.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Harsha,
> > >
> > > I mentioned policies and the authorizer. For example, with
> > > CreateTopicPolicy, you can implement the limits you describe. If you have
> > > ideas of how that should be improved, please submit a KIP. My point is
> > that
> > > this KIP is not introducing any new functionality with regards to what
> > > rogue clients can do. It's using the existing protocol that is already
> > > exposed via the AdminClient. So, I don't think we need to address it in
> > > this KIP. Does that make sense?
> > >
> > > Ismael
> > >
> > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > Ismael,
> > > Sure AdminClient can do that and we should've shipped a config or use the
> > > existing one to block that. Not all users are yet to upgrade to
> > AdminClient
> > > and start using that to cause issues yet. In shared environment we should
> > > allow server to set sane defaults and not allow every client to go ahead
> > > create random no.of topic/partitions and replication factor. Even if the
> > > users want to allow topic creation proposed in the KIP , it makes sense
> > to
> > > have some guards against the no.of partitions and replication factor.
> > > Authorizer is not always an answer to block requests and having users set
> > > server side configs to protect a multi-tenant environment is required.
> > In a
> > > non-secure environment Authorizer is a blunt instrument either you end up
> > > blocking everyone or allowing everyone.
> > > I am asking to have server side that allow clients to create topics or
> > not
> > > , if they are allowed set a ceiling on max no.of partitions and
> > > replication-factor.
> > >
> > > -Harsha
> > >
> > > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> > >
> > > Harsha,
> > >
> > > Rogue clients can use the admin client to create topics and partitions.
> > > ACLs and policies can help in that case as well as this one.
> > >
> > > Ismael
> > >
> > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > > Thanks for the KIP.
> > > "When server-side auto-creation is disabled, client-side auto-creation
> > > will try to use client-side configurations"
> > > If I understand correctly, this KIP is removing any server-side blocking
> > > client auto creation of topic?
> > > if so this will present potential issue of rogue client creating ton of
> > > topic-partitions and potentially bringing down the service for everyone
> > >
> > > or
> > >
> > > degrade the service itself.
> > > By reading the KIP its not clear to me that there is a clear way to block
> > > auto creation topics of all together from clients by server side config.
> > > Server side configs of default topic, partitions should take higher
> > > precedence and client shouldn't be able to create a topic with higher
> > >
> > > no.of
> > >
> > > partitions, replication than what server config specifies.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> > > wrote:
> > >
> > > Hi all,
> > > I made some changes to the KIP. Hopefully this configuration change
> > >
> > > will
> > >
> > > make things a little clearer.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > >
> > > Please let me know if you have any feedback or questions!
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think you bring up a good point. It would be better if we didn't ever
> > > have to set up client-side configuration for this feature, and KIP-464
> > > would let us skip this entirely.
> > >
> > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > >
> > > Hi Mickael,
> > > I agree that KIP-464 works on newer brokers, but I was a bit worried
> > >
> > > how
> > >
> > > things would play out on older brokers that* do not *have KIP 464
> > >
> > > included.
> > >
> > > Is it enough to throw an error in this case when producer configs are
> > >
> > > not
> > >
> > > specified?
> > >
> > > I think the right thing to do would be to log an error message in the
> > > client. We will need to have that capability in any case, to cover
> > > scenarios like the client trying to auto-create a topic that they don't
> > > have permission to create. Or a client trying to create a topic on a
> > >
> > > broker
> > >
> > > so old that CreateTopicsRequest is not supported.
> > >
> > > The big downside to relying on KIP-464 is that it is a very recent
> > >
> > > feature
> > >
> > > -- so recent that it hasn't even made its way to any official Apache
> > > release. It's scheduled for the upcoming 2.4 release in a few months.
> > >
> > > So if you view this KIP as a step towards removing broker-side
> > > auto-create, you might want to support older brokers just to accelerate
> > > adoption, and hasten the day when we can finally flip broker-side
> > > auto-create to off (or even remove it entirely).
> > >
> > > I have to agree, though, that having client-side configurations for
> > >
> > > number
> > >
> > > of partitions and replication factor is messy. Maybe it would be worth
> > >
> > > it
> > >
> > > to restrict support to post-KIP-464 brokers, if we could avoid creating
> > > more configs.
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > We can rely on KIP-464 which allows to omit the partition count or
> > > replication factor when creating a topic. In that case, the broker
> > >
> > > defaults
> > >
> > > are used.
> > >
> > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > > wrote:
> > >
> > > Michael,
> > > That makes sense to me!
> > > To clarify, in the current state of the KIP, the producer does not
> > >
> > > rely
> > >
> > > on
> > >
> > > the broker to autocreate--if the broker's config is disabled, then
> > >
> > > the
> > >
> > > producer can autocreate on its own with a create topic request (the
> > >
> > > same
> > >
> > > type of request the admin client uses).
> > > However, if both configs are enabled, the broker will autocreate
> > >
> > > through
> > >
> > > a
> > >
> > > metadata request before the producer gets a chance. Of course, the way
> > >
> > > to
> > >
> > > avoid this, is to do as you suggested, and set
> > >
> > > the
> > >
> > > "allow_auto_topic_creation" field to false.
> > >
> > > I think the only thing we need to be careful with in this setup is
> > >
> > > without
> > >
> > > KIP 464, we can not use broker defaults for this topic. A user needs
> > >
> > > to
> > >
> > > specify the number of partition and replication factor in the config.
> > >
> > > An
> > >
> > > alternative to this is to have coded defaults for when these
> > >
> > > configs
> > >
> > > are
> > >
> > > unspecified, but it is not immediately apparent what these defaults
> > >
> > > should
> > >
> > > be.
> > >
> > > Thanks again for reading my KIP,
> > > Justine
> > >
> > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the response!
> > > In my opinion, it would be better if the producer did not rely at
> > >
> > > all
> > >
> > > on the broker auto create feature as this is what we're aiming to
> > > deprecate. When requesting metadata we can set the
> > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > creation. Then if the topic is not existing, send a CreateTopicRequest.
> > >
> > > What do you think?
> > >
> > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Currently the way it is implemented, the broker auto-creation
> > >
> > > configuration
> > >
> > > takes precedence. The producer will not use the CreateTopics
> > >
> > > request.
> > >
> > > (Technically it can--but the topic will already be created
> > >
> > > through
> > >
> > > the
> > >
> > > broker, so it will never try to create the topic.) It is possible to
> > > change this however, and I'd be happy to
> > >
> > > discuss
> > >
> > > the
> > >
> > > benefits of this alternative.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP!
> > >
> > > In case auto creation is enabled on both the client and server,
> > >
> > > will
> > >
> > > the producer still use the AdminClient (CreateTopics request)
> > >
> > > to
> > >
> > > create topics? and not rely on the broker auto create. I'm guessing the
> > > answer is yes but can you make it explicit.
> > >
> > > Thank you
> > >
> > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi,
> > > Just a friendly reminder to take a look at this KIP if you
> > >
> > > have
> > >
> > > the
> > >
> > > time.
> > >
> > > I was thinking about broker vs. client default precedence,
> > >
> > > and I
> > >
> > > think it
> > >
> > > makes sense to keep the broker as the default used when both
> > >
> > > client-side
> > >
> > > and broker-side defaults are configured. The idea is that
> > >
> > > there
> > >
> > > would be
> > >
> > > pretty clear documentation, and that many systems with
> > >
> > > configurations
> > >
> > > that
> > >
> > > the client could not change would likely have the auto-create
> > >
> > > default
> > >
> > > off.
> > >
> > > (In cloud for example).
> > >
> > > It also seems like in most cases, the consumer config
> > > 'allow.auto.create.topics' was created to actually prevent
> > >
> > > the
> > >
> > > creation
> > >
> > > of
> > >
> > > topics, so the loss of creation functionality will not be a
> > >
> > > big
> > >
> > > problem.
> > >
> > > I'm happy to discuss any other compatibility problems or
> > >
> > > components
> > >
> > > of
> > >
> > > this KIP.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > >
> > > jolshan@confluent.io
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I was looking at this KIP again, and there is a decision I
> > >
> > > made
> > >
> > > that I
> > >
> > > think is worth discussing.
> > >
> > > In the case where both the broker and producer's
> > > 'auto.create.topics.enable' are set to true, we have to
> > >
> > > choose
> > >
> > > either
> > >
> > > the
> > >
> > > broker configs or the producer configs for the replication
> > > factor/partitions.
> > >
> > > Currently, the decision is to have the broker defaults take
> > >
> > > precedence.
> > >
> > > (It is easier to do this in the implementation.) It also
> > >
> > > makes
> > >
> > > some
> > >
> > > sense
> > >
> > > for this behavior to take precedence since this behavior
> > >
> > > already
> > >
> > > occurs as
> > >
> > > the default.
> > >
> > > However, I was wondering if it would be odd for those who
> > >
> > > can
> > >
> > > only
> > >
> > > see
> > >
> > > the
> > >
> > > producer side to set configs for replication
> > >
> > > factor/partitions
> > >
> > > and
> > >
> > > see
> > >
> > > different results. Currently the documentation for the
> > >
> > > config
> > >
> > > states
> > >
> > > that
> > >
> > > the config values are only used when the broker config is
> > >
> > > not
> > >
> > > enabled,
> > >
> > > but
> > >
> > > this might not always be clear to the user. Changing the
> > >
> > > code
> > >
> > > to
> > >
> > > have
> > >
> > > the
> > >
> > > producer's configurations take precedence is possible, but
> > >
> > > I
> > >
> > > was
> > >
> > > wondering
> > >
> > > what everyone thought.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Just a quick update--
> > >
> > > It seems that enabling both the broker and producer
> > >
> > > configs
> > >
> > > works
> > >
> > > fine,
> > >
> > > except that the broker configurations for partitions,
> > >
> > > replication
> > >
> > > factor
> > >
> > > take precedence.
> > > I don't know if that is something we would want to
> > >
> > > change, but
> > >
> > > I'll be
> > >
> > > updating the KIP for now to reflect this. Perhaps we would
> > >
> > > want to
> > >
> > > add more
> > >
> > > to the documentation of the the producer configs to
> > >
> > > clarify.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for looking at the KIP. I can definitely add to
> > >
> > > the
> > >
> > > title
> > >
> > > to
> > >
> > > make
> > >
> > > it more clear.
> > >
> > > It makes sense that both configurations could be turned
> > >
> > > on
> > >
> > > since
> > >
> > > there
> > >
> > > are many cases where the user can not control the
> > >
> > > server-side
> > >
> > > configurations. I was a little concerned about how both
> > >
> > > interacting
> > >
> > > would
> > >
> > > work out -- if there would be to many requests for new
> > >
> > > topics,
> > >
> > > for
> > >
> > > example.
> > >
> > > But it since it does make sense to allow both
> > >
> > > configurations
> > >
> > > enabled, I
> > >
> > > will test out some scenarios and I'll change the KIP to
> > >
> > > support
> > >
> > > this.
> > >
> > > I also agree with documentation about distinguishing the
> > >
> > > differences. I
> > >
> > > was having some trouble with the wording but I like the
> > >
> > > phrases
> > >
> > > "server-side" and "client-side." That's a good
> > >
> > > distinction I
> > >
> > > can
> > >
> > > use
> > >
> > > when
> > >
> > > describing.
> > >
> > > I'll try to update the KIP soon keeping everyone's input
> > >
> > > in
> > >
> > > mind.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > >
> > > cmccabe@apache.org
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP. This seems like a good step towards
> > >
> > > removing
> > >
> > > server-side topic auto-creation.
> > >
> > > We should add included "client-side" to the title of
> > >
> > > the KIP
> > >
> > > somewhere,
> > >
> > > to make it clear that we're talking about client-side
> > >
> > > auto
> > >
> > > creation.
> > >
> > > The KIP says:
> > >
> > > In order to automatically create topics with the
> > >
> > > producer, the
> > >
> > > producer's
> > >
> > > auto.create.topics.enable config must be set to true
> > >
> > > and
> > >
> > > the
> > >
> > > broker
> > >
> > > config should be set to false
> > >
> > > From a user's point of view, this seems
> > >
> > > counter-intuitive.
> > >
> > > In
> > >
> > > order to
> > >
> > > auto-create topics the broker's
> > >
> > > auto.create.topics.enable
> > >
> > > config
> > >
> > > should be
> > >
> > > set to false? It seems like the server-side
> > >
> > > auto-create is
> > >
> > > unrelated to
> > >
> > > the client-side auto-create. We could have both turned
> > >
> > > on
> > >
> > > (and
> > >
> > > I'm
> > >
> > > sure
> > >
> > > that in the real world, people will try this
> > >
> > > configuration...)
> > >
> > > There's no
> > >
> > > reason not to support this, I think.
> > >
> > > We should add some documentation explaining the
> > >
> > > difference
> > >
> > > between
> > >
> > > server-side and client-side auto-creation. Without
> > >
> > > documentation,
> > >
> > > an admin
> > >
> > > might think that they had disabled all forms of
> > >
> > > auto-creation by
> > >
> > > setting
> > >
> > > the -side setting to false-- but this is not the case,
> > >
> > > of
> > >
> > > course.
> > >
> > > best,
> > > Colin
> > >
> > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > >
> > > Hi Dhruvil,
> > >
> > > Thanks for reading the KIP!
> > > That was the general idea for deprecation. We would
> > >
> > > log a
> > >
> > > warning
> > >
> > > when the
> > >
> > > config is enabled on the broker.
> > > I also believe that there would be a change to
> > >
> > > documentation.
> > >
> > > If there is anything else that should be done, please
> > >
> > > let
> > >
> > > me
> > >
> > > know!
> > >
> > > Justine
> > >
> > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > >
> > > dhruvil@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP, this is great!
> > >
> > > Could you add some more information about what
> > >
> > > deprecating
> > >
> > > the
> > >
> > > broker
> > >
> > > configuration means? Would we log a warning in the
> > >
> > > logs
> > >
> > > when
> > >
> > > auto
> > >
> > > topic
> > >
> > > creation is enabled on the broker, for example?
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > > deprecate the current system of
> > >
> > > auto-creating
> > >
> > > topics
> > >
> > > through requests to the metadata and give the
> > >
> > > producer the
> > >
> > > ability to
> > >
> > > automatically create topics instead.
> > >
> > > More information can be found here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > >
> > > Thank you,
> > > Justine Olshan
> > >
> > >
> >

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Justine Olshan <jo...@confluent.io>.

Hi Harsha,

After taking this all into consideration, I've updated the KIP to no longer
allow client-side configuration of replication factor and partitions.
Instead, the broker defaults will be used as long as the broker supports
KIP 464.
If the broker does not support this KIP, then the client can not create
topics on its own. (Behavior that exists now)

I think this will help with your concerns. Please let me know if you
further feedback.

Thank you,
Justine

On Tue, Aug 6, 2019 at 10:49 AM Harsha Chintalapani <ka...@harsha.io> wrote:

> Hi,
>     Even with policies one needs to implement that, so for every user who
> doesn't want a producer to create topics or have limits around partitions &
> replication factor they have to implement a policy.
>       The KIP is changing the behavior , it might not be introducing the
> new functionality but it will enable producers to override the create topic
> config settings on the broker. What I am asking for to provide a config
> that will disable auto creation of topics and if its enabled set some sane
> defaults so that clients can create a topic with in those limits. I don't
> see how this not related to this KIP.
>      If the server config options as I suggested doesn't interest you at
> least have a default CreateTopicPolices in place.
>        To give an example, In our environment we disable the
> auto.create.topic.enable and force users to go through a centralized
> service as we want capture more details about the topic creation and
> requirements. With this KIP, a producer can create a topic with no bounds.
>  Another example max.message.size we define that at cluster level and one
> can override max.messsage.size at topic level, any misconfiguration at this
> will cause service degradation.  Its not always about the rogue clients,
> users can easily misconfigure and can cause an outage.
> Again we can talk about CreateTopicPolicy but without having a default
> implementation and asking everyone to implement their own while changing
> the behavior in producer  doesn't make sense to me.
>
> Thanks,
> Harsha
>
>
> On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Harsha,
> >
> > I mentioned policies and the authorizer. For example, with
> > CreateTopicPolicy, you can implement the limits you describe. If you have
> > ideas of how that should be improved, please submit a KIP. My point is
> that
> > this KIP is not introducing any new functionality with regards to what
> > rogue clients can do. It's using the existing protocol that is already
> > exposed via the AdminClient. So, I don't think we need to address it in
> > this KIP. Does that make sense?
> >
> > Ismael
> >
> > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > Ismael,
> > Sure AdminClient can do that and we should've shipped a config or use the
> > existing one to block that. Not all users are yet to upgrade to
> AdminClient
> > and start using that to cause issues yet. In shared environment we should
> > allow server to set sane defaults and not allow every client to go ahead
> > create random no.of topic/partitions and replication factor. Even if the
> > users want to allow topic creation proposed in the KIP , it makes sense
> to
> > have some guards against the no.of partitions and replication factor.
> > Authorizer is not always an answer to block requests and having users set
> > server side configs to protect a multi-tenant environment is required.
> In a
> > non-secure environment Authorizer is a blunt instrument either you end up
> > blocking everyone or allowing everyone.
> > I am asking to have server side that allow clients to create topics or
> not
> > , if they are allowed set a ceiling on max no.of partitions and
> > replication-factor.
> >
> > -Harsha
> >
> > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> >
> > Harsha,
> >
> > Rogue clients can use the admin client to create topics and partitions.
> > ACLs and policies can help in that case as well as this one.
> >
> > Ismael
> >
> > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> >
> > wrote:
> >
> > Hi Justine,
> > Thanks for the KIP.
> > "When server-side auto-creation is disabled, client-side auto-creation
> > will try to use client-side configurations"
> > If I understand correctly, this KIP is removing any server-side blocking
> > client auto creation of topic?
> > if so this will present potential issue of rogue client creating ton of
> > topic-partitions and potentially bringing down the service for everyone
> >
> > or
> >
> > degrade the service itself.
> > By reading the KIP its not clear to me that there is a clear way to block
> > auto creation topics of all together from clients by server side config.
> > Server side configs of default topic, partitions should take higher
> > precedence and client shouldn't be able to create a topic with higher
> >
> > no.of
> >
> > partitions, replication than what server config specifies.
> >
> > Thanks,
> > Harsha
> >
> > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> > wrote:
> >
> > Hi all,
> > I made some changes to the KIP. Hopefully this configuration change
> >
> > will
> >
> > make things a little clearer.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> >
> > Please let me know if you have any feedback or questions!
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> >
> > wrote:
> >
> > Hi Mickael,
> >
> > I think you bring up a good point. It would be better if we didn't ever
> > have to set up client-side configuration for this feature, and KIP-464
> > would let us skip this entirely.
> >
> > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> >
> > Hi Mickael,
> > I agree that KIP-464 works on newer brokers, but I was a bit worried
> >
> > how
> >
> > things would play out on older brokers that* do not *have KIP 464
> >
> > included.
> >
> > Is it enough to throw an error in this case when producer configs are
> >
> > not
> >
> > specified?
> >
> > I think the right thing to do would be to log an error message in the
> > client. We will need to have that capability in any case, to cover
> > scenarios like the client trying to auto-create a topic that they don't
> > have permission to create. Or a client trying to create a topic on a
> >
> > broker
> >
> > so old that CreateTopicsRequest is not supported.
> >
> > The big downside to relying on KIP-464 is that it is a very recent
> >
> > feature
> >
> > -- so recent that it hasn't even made its way to any official Apache
> > release. It's scheduled for the upcoming 2.4 release in a few months.
> >
> > So if you view this KIP as a step towards removing broker-side
> > auto-create, you might want to support older brokers just to accelerate
> > adoption, and hasten the day when we can finally flip broker-side
> > auto-create to off (or even remove it entirely).
> >
> > I have to agree, though, that having client-side configurations for
> >
> > number
> >
> > of partitions and replication factor is messy. Maybe it would be worth
> >
> > it
> >
> > to restrict support to post-KIP-464 brokers, if we could avoid creating
> > more configs.
> >
> > best,
> > Colin
> >
> > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> >
> > mickael.maison@gmail.com
> >
> > wrote:
> >
> > Hi Justine,
> >
> > We can rely on KIP-464 which allows to omit the partition count or
> > replication factor when creating a topic. In that case, the broker
> >
> > defaults
> >
> > are used.
> >
> > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > wrote:
> >
> > Michael,
> > That makes sense to me!
> > To clarify, in the current state of the KIP, the producer does not
> >
> > rely
> >
> > on
> >
> > the broker to autocreate--if the broker's config is disabled, then
> >
> > the
> >
> > producer can autocreate on its own with a create topic request (the
> >
> > same
> >
> > type of request the admin client uses).
> > However, if both configs are enabled, the broker will autocreate
> >
> > through
> >
> > a
> >
> > metadata request before the producer gets a chance. Of course, the way
> >
> > to
> >
> > avoid this, is to do as you suggested, and set
> >
> > the
> >
> > "allow_auto_topic_creation" field to false.
> >
> > I think the only thing we need to be careful with in this setup is
> >
> > without
> >
> > KIP 464, we can not use broker defaults for this topic. A user needs
> >
> > to
> >
> > specify the number of partition and replication factor in the config.
> >
> > An
> >
> > alternative to this is to have coded defaults for when these
> >
> > configs
> >
> > are
> >
> > unspecified, but it is not immediately apparent what these defaults
> >
> > should
> >
> > be.
> >
> > Thanks again for reading my KIP,
> > Justine
> >
> > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> >
> > mickael.maison@gmail.com
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the response!
> > In my opinion, it would be better if the producer did not rely at
> >
> > all
> >
> > on the broker auto create feature as this is what we're aiming to
> > deprecate. When requesting metadata we can set the
> > "allow_auto_topic_creation" field to false to avoid the broker auto
> > creation. Then if the topic is not existing, send a CreateTopicRequest.
> >
> > What do you think?
> >
> > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Currently the way it is implemented, the broker auto-creation
> >
> > configuration
> >
> > takes precedence. The producer will not use the CreateTopics
> >
> > request.
> >
> > (Technically it can--but the topic will already be created
> >
> > through
> >
> > the
> >
> > broker, so it will never try to create the topic.) It is possible to
> > change this however, and I'd be happy to
> >
> > discuss
> >
> > the
> >
> > benefits of this alternative.
> >
> > Thank you,
> > Justine
> >
> > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> >
> > mickael.maison@gmail.com>
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP!
> >
> > In case auto creation is enabled on both the client and server,
> >
> > will
> >
> > the producer still use the AdminClient (CreateTopics request)
> >
> > to
> >
> > create topics? and not rely on the broker auto create. I'm guessing the
> > answer is yes but can you make it explicit.
> >
> > Thank you
> >
> > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hi,
> > Just a friendly reminder to take a look at this KIP if you
> >
> > have
> >
> > the
> >
> > time.
> >
> > I was thinking about broker vs. client default precedence,
> >
> > and I
> >
> > think it
> >
> > makes sense to keep the broker as the default used when both
> >
> > client-side
> >
> > and broker-side defaults are configured. The idea is that
> >
> > there
> >
> > would be
> >
> > pretty clear documentation, and that many systems with
> >
> > configurations
> >
> > that
> >
> > the client could not change would likely have the auto-create
> >
> > default
> >
> > off.
> >
> > (In cloud for example).
> >
> > It also seems like in most cases, the consumer config
> > 'allow.auto.create.topics' was created to actually prevent
> >
> > the
> >
> > creation
> >
> > of
> >
> > topics, so the loss of creation functionality will not be a
> >
> > big
> >
> > problem.
> >
> > I'm happy to discuss any other compatibility problems or
> >
> > components
> >
> > of
> >
> > this KIP.
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> >
> > jolshan@confluent.io
> >
> > wrote:
> >
> > Hello all,
> >
> > I was looking at this KIP again, and there is a decision I
> >
> > made
> >
> > that I
> >
> > think is worth discussing.
> >
> > In the case where both the broker and producer's
> > 'auto.create.topics.enable' are set to true, we have to
> >
> > choose
> >
> > either
> >
> > the
> >
> > broker configs or the producer configs for the replication
> > factor/partitions.
> >
> > Currently, the decision is to have the broker defaults take
> >
> > precedence.
> >
> > (It is easier to do this in the implementation.) It also
> >
> > makes
> >
> > some
> >
> > sense
> >
> > for this behavior to take precedence since this behavior
> >
> > already
> >
> > occurs as
> >
> > the default.
> >
> > However, I was wondering if it would be odd for those who
> >
> > can
> >
> > only
> >
> > see
> >
> > the
> >
> > producer side to set configs for replication
> >
> > factor/partitions
> >
> > and
> >
> > see
> >
> > different results. Currently the documentation for the
> >
> > config
> >
> > states
> >
> > that
> >
> > the config values are only used when the broker config is
> >
> > not
> >
> > enabled,
> >
> > but
> >
> > this might not always be clear to the user. Changing the
> >
> > code
> >
> > to
> >
> > have
> >
> > the
> >
> > producer's configurations take precedence is possible, but
> >
> > I
> >
> > was
> >
> > wondering
> >
> > what everyone thought.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Just a quick update--
> >
> > It seems that enabling both the broker and producer
> >
> > configs
> >
> > works
> >
> > fine,
> >
> > except that the broker configurations for partitions,
> >
> > replication
> >
> > factor
> >
> > take precedence.
> > I don't know if that is something we would want to
> >
> > change, but
> >
> > I'll be
> >
> > updating the KIP for now to reflect this. Perhaps we would
> >
> > want to
> >
> > add more
> >
> > to the documentation of the the producer configs to
> >
> > clarify.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hi Colin,
> >
> > Thanks for looking at the KIP. I can definitely add to
> >
> > the
> >
> > title
> >
> > to
> >
> > make
> >
> > it more clear.
> >
> > It makes sense that both configurations could be turned
> >
> > on
> >
> > since
> >
> > there
> >
> > are many cases where the user can not control the
> >
> > server-side
> >
> > configurations. I was a little concerned about how both
> >
> > interacting
> >
> > would
> >
> > work out -- if there would be to many requests for new
> >
> > topics,
> >
> > for
> >
> > example.
> >
> > But it since it does make sense to allow both
> >
> > configurations
> >
> > enabled, I
> >
> > will test out some scenarios and I'll change the KIP to
> >
> > support
> >
> > this.
> >
> > I also agree with documentation about distinguishing the
> >
> > differences. I
> >
> > was having some trouble with the wording but I like the
> >
> > phrases
> >
> > "server-side" and "client-side." That's a good
> >
> > distinction I
> >
> > can
> >
> > use
> >
> > when
> >
> > describing.
> >
> > I'll try to update the KIP soon keeping everyone's input
> >
> > in
> >
> > mind.
> >
> > Thanks,
> > Justine
> >
> > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> >
> > cmccabe@apache.org
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP. This seems like a good step towards
> >
> > removing
> >
> > server-side topic auto-creation.
> >
> > We should add included "client-side" to the title of
> >
> > the KIP
> >
> > somewhere,
> >
> > to make it clear that we're talking about client-side
> >
> > auto
> >
> > creation.
> >
> > The KIP says:
> >
> > In order to automatically create topics with the
> >
> > producer, the
> >
> > producer's
> >
> > auto.create.topics.enable config must be set to true
> >
> > and
> >
> > the
> >
> > broker
> >
> > config should be set to false
> >
> > From a user's point of view, this seems
> >
> > counter-intuitive.
> >
> > In
> >
> > order to
> >
> > auto-create topics the broker's
> >
> > auto.create.topics.enable
> >
> > config
> >
> > should be
> >
> > set to false? It seems like the server-side
> >
> > auto-create is
> >
> > unrelated to
> >
> > the client-side auto-create. We could have both turned
> >
> > on
> >
> > (and
> >
> > I'm
> >
> > sure
> >
> > that in the real world, people will try this
> >
> > configuration...)
> >
> > There's no
> >
> > reason not to support this, I think.
> >
> > We should add some documentation explaining the
> >
> > difference
> >
> > between
> >
> > server-side and client-side auto-creation. Without
> >
> > documentation,
> >
> > an admin
> >
> > might think that they had disabled all forms of
> >
> > auto-creation by
> >
> > setting
> >
> > the -side setting to false-- but this is not the case,
> >
> > of
> >
> > course.
> >
> > best,
> > Colin
> >
> > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> >
> > Hi Dhruvil,
> >
> > Thanks for reading the KIP!
> > That was the general idea for deprecation. We would
> >
> > log a
> >
> > warning
> >
> > when the
> >
> > config is enabled on the broker.
> > I also believe that there would be a change to
> >
> > documentation.
> >
> > If there is anything else that should be done, please
> >
> > let
> >
> > me
> >
> > know!
> >
> > Justine
> >
> > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> >
> > dhruvil@confluent.io>
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP, this is great!
> >
> > Could you add some more information about what
> >
> > deprecating
> >
> > the
> >
> > broker
> >
> > configuration means? Would we log a warning in the
> >
> > logs
> >
> > when
> >
> > auto
> >
> > topic
> >
> > creation is enabled on the broker, for example?
> >
> > Thanks,
> > Dhruvil
> >
> > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hello all,
> >
> > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > deprecate the current system of
> >
> > auto-creating
> >
> > topics
> >
> > through requests to the metadata and give the
> >
> > producer the
> >
> > ability to
> >
> > automatically create topics instead.
> >
> > More information can be found here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> >
> > Thank you,
> > Justine Olshan
> >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Harsha Chintalapani <ka...@harsha.io>.

Hi Justin,
              Thanks for making changes. I still have concern that we are
prioritizing producer config over server side which is breaking the
backward compatibility of broker's auto.topic.create.enable as far as
producer is concerned.
           Also
https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
only
allows creation of topic if both allow.auto.create.topics set to true
and auto.create.topics.enable
on the broker side set to true.  This KIP is changing that contract for
producers and allow them to take control  even broker side is turned off.
Why can't we have the same behavior.  You can still achieve the goal for
this KIP by allowing auto.create.topics.enable to take precedence and have
a similar behavior to that of consumer.


Thanks,
Harsha


On Tue, Aug 06, 2019 at 6:06 PM, Harsha Ch <ha...@gmail.com> wrote:

> Hi Colin,
>          There is no behavior in Kafka producer that allowed users to
> delete the topics or delete the records. So
> citing them as an example doesn't makes sense in this context. But there
> is a functionality which allowed creation of topics if they don't exist in
> the cluster and this behavior could be controlled by a config on the server
> side. Now with this KIP we are allowing producer to make that decision
> without any gateway on the server via configs. Any user who is not aware of
> such changes
> can accidentally create these topics and we are essentially removing a
> config that exists in brokers today to block this accidental creation and
> allowing clients to take control.
>       Not sure how the AdminClient applies here, It is an external API and
> is not part of KafkaProducer so any user who updates to latest version of
> Kafka with this feature get to create the topics.
> They have to build a tooling around AdminClient allow themselves to create
> topics.
>       I still didn't get any positives of not having server side configs?
> if you want to turn them on and allow any client to create topics set the
> default to true
> and allow users who doesn't want to have this feature let them turn them
> off. It will be the exact behavior as it is today, as far as producer is
> concerned. I am not
> understanding why not having server side configs to gateways such a hard
> requirement and this behavior exists today. As far I am concerned this
> breaks the backward compatibility.
>
> Thanks,
> Harsha
>
>
> On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe <cm...@apache.org> wrote:
>
> Hi Harsha,
>
> Thanks for taking a look.
>
> I'm not sure I follow the discussion about AdminClient.  KafkaAdminClient
> has been around for about 2 years at this point as a public class.  There
> are many programs that use it to automatically create topics.  Kafka
> Streams does this, for example.  If any of your users run Kafka Streams,
> they will be auto-creating topics, regardless of what setting you use for
> auto.create.topics.enable.
>
> A big part of the annoyance of auto-topic creation right now is that it is
> on by default.  The new configuration proposed by KIP-487 wouldn't be.
> Users would have to explicitly opt in to the new behavior of client-side
> topic creation.  If you run without security, you're already putting a huge
> amount of trust in your users.  For example, you trust them not to delete
> records with the kafka-delete-records.sh command, or delete topics with
> kafka-topics.sh.  Trusting them not to set a certain config value seems
> minor in comparison, right?
>
> best,
> Colin
>
> On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > Hi,
> >     Even with policies one needs to implement that, so for every user who
> > doesn't want a producer to create topics or have limits around
> partitions &
> > replication factor they have to implement a policy.
> >       The KIP is changing the behavior , it might not be introducing the
> > new functionality but it will enable producers to override the create
> topic
> > config settings on the broker. What I am asking for to provide a config
> > that will disable auto creation of topics and if its enabled set some
> sane
> > defaults so that clients can create a topic with in those limits. I don't
> > see how this not related to this KIP.
> >      If the server config options as I suggested doesn't interest you at
> > least have a default CreateTopicPolices in place.
> >        To give an example, In our environment we disable the
> > auto.create.topic.enable and force users to go through a centralized
> > service as we want capture more details about the topic creation and
> > requirements. With this KIP, a producer can create a topic with no
> bounds.
> >  Another example max.message.size we define that at cluster level and one
> > can override max.messsage.size at topic level, any misconfiguration at
> this
> > will cause service degradation.  Its not always about the rogue clients,
> > users can easily misconfigure and can cause an outage.
> > Again we can talk about CreateTopicPolicy but without having a default
> > implementation and asking everyone to implement their own while changing
> > the behavior in producer  doesn't make sense to me.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Harsha,
> > >
> > > I mentioned policies and the authorizer. For example, with
> > > CreateTopicPolicy, you can implement the limits you describe. If you
> have
> > > ideas of how that should be improved, please submit a KIP. My point is
> that
> > > this KIP is not introducing any new functionality with regards to what
> > > rogue clients can do. It's using the existing protocol that is already
> > > exposed via the AdminClient. So, I don't think we need to address it in
> > > this KIP. Does that make sense?
> > >
> > > Ismael
> > >
> > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > Ismael,
> > > Sure AdminClient can do that and we should've shipped a config or use
> the
> > > existing one to block that. Not all users are yet to upgrade to
> AdminClient
> > > and start using that to cause issues yet. In shared environment we
> should
> > > allow server to set sane defaults and not allow every client to go
> ahead
> > > create random no.of topic/partitions and replication factor. Even if
> the
> > > users want to allow topic creation proposed in the KIP , it makes
> sense to
> > > have some guards against the no.of partitions and replication factor.
> > > Authorizer is not always an answer to block requests and having users
> set
> > > server side configs to protect a multi-tenant environment is required.
> In a
> > > non-secure environment Authorizer is a blunt instrument either you end
> up
> > > blocking everyone or allowing everyone.
> > > I am asking to have server side that allow clients to create topics or
> not
> > > , if they are allowed set a ceiling on max no.of partitions and
> > > replication-factor.
> > >
> > > -Harsha
> > >
> > > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> > >
> > > Harsha,
> > >
> > > Rogue clients can use the admin client to create topics and partitions.
> > > ACLs and policies can help in that case as well as this one.
> > >
> > > Ismael
> > >
> > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > > Thanks for the KIP.
> > > "When server-side auto-creation is disabled, client-side auto-creation
> > > will try to use client-side configurations"
> > > If I understand correctly, this KIP is removing any server-side
> blocking
> > > client auto creation of topic?
> > > if so this will present potential issue of rogue client creating ton of
> > > topic-partitions and potentially bringing down the service for everyone
> > >
> > > or
> > >
> > > degrade the service itself.
> > > By reading the KIP its not clear to me that there is a clear way to
> block
> > > auto creation topics of all together from clients by server side
> config.
> > > Server side configs of default topic, partitions should take higher
> > > precedence and client shouldn't be able to create a topic with higher
> > >
> > > no.of
> > >
> > > partitions, replication than what server config specifies.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> > > wrote:
> > >
> > > Hi all,
> > > I made some changes to the KIP. Hopefully this configuration change
> > >
> > > will
> > >
> > > make things a little clearer.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > >
> > > Please let me know if you have any feedback or questions!
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think you bring up a good point. It would be better if we didn't ever
> > > have to set up client-side configuration for this feature, and KIP-464
> > > would let us skip this entirely.
> > >
> > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > >
> > > Hi Mickael,
> > > I agree that KIP-464 works on newer brokers, but I was a bit worried
> > >
> > > how
> > >
> > > things would play out on older brokers that* do not *have KIP 464
> > >
> > > included.
> > >
> > > Is it enough to throw an error in this case when producer configs are
> > >
> > > not
> > >
> > > specified?
> > >
> > > I think the right thing to do would be to log an error message in the
> > > client. We will need to have that capability in any case, to cover
> > > scenarios like the client trying to auto-create a topic that they don't
> > > have permission to create. Or a client trying to create a topic on a
> > >
> > > broker
> > >
> > > so old that CreateTopicsRequest is not supported.
> > >
> > > The big downside to relying on KIP-464 is that it is a very recent
> > >
> > > feature
> > >
> > > -- so recent that it hasn't even made its way to any official Apache
> > > release. It's scheduled for the upcoming 2.4 release in a few months.
> > >
> > > So if you view this KIP as a step towards removing broker-side
> > > auto-create, you might want to support older brokers just to accelerate
> > > adoption, and hasten the day when we can finally flip broker-side
> > > auto-create to off (or even remove it entirely).
> > >
> > > I have to agree, though, that having client-side configurations for
> > >
> > > number
> > >
> > > of partitions and replication factor is messy. Maybe it would be worth
> > >
> > > it
> > >
> > > to restrict support to post-KIP-464 brokers, if we could avoid creating
> > > more configs.
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > We can rely on KIP-464 which allows to omit the partition count or
> > > replication factor when creating a topic. In that case, the broker
> > >
> > > defaults
> > >
> > > are used.
> > >
> > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > > wrote:
> > >
> > > Michael,
> > > That makes sense to me!
> > > To clarify, in the current state of the KIP, the producer does not
> > >
> > > rely
> > >
> > > on
> > >
> > > the broker to autocreate--if the broker's config is disabled, then
> > >
> > > the
> > >
> > > producer can autocreate on its own with a create topic request (the
> > >
> > > same
> > >
> > > type of request the admin client uses).
> > > However, if both configs are enabled, the broker will autocreate
> > >
> > > through
> > >
> > > a
> > >
> > > metadata request before the producer gets a chance. Of course, the way
> > >
> > > to
> > >
> > > avoid this, is to do as you suggested, and set
> > >
> > > the
> > >
> > > "allow_auto_topic_creation" field to false.
> > >
> > > I think the only thing we need to be careful with in this setup is
> > >
> > > without
> > >
> > > KIP 464, we can not use broker defaults for this topic. A user needs
> > >
> > > to
> > >
> > > specify the number of partition and replication factor in the config.
> > >
> > > An
> > >
> > > alternative to this is to have coded defaults for when these
> > >
> > > configs
> > >
> > > are
> > >
> > > unspecified, but it is not immediately apparent what these defaults
> > >
> > > should
> > >
> > > be.
> > >
> > > Thanks again for reading my KIP,
> > > Justine
> > >
> > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the response!
> > > In my opinion, it would be better if the producer did not rely at
> > >
> > > all
> > >
> > > on the broker auto create feature as this is what we're aiming to
> > > deprecate. When requesting metadata we can set the
> > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > creation. Then if the topic is not existing, send a CreateTopicRequest.
> > >
> > > What do you think?
> > >
> > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Currently the way it is implemented, the broker auto-creation
> > >
> > > configuration
> > >
> > > takes precedence. The producer will not use the CreateTopics
> > >
> > > request.
> > >
> > > (Technically it can--but the topic will already be created
> > >
> > > through
> > >
> > > the
> > >
> > > broker, so it will never try to create the topic.) It is possible to
> > > change this however, and I'd be happy to
> > >
> > > discuss
> > >
> > > the
> > >
> > > benefits of this alternative.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP!
> > >
> > > In case auto creation is enabled on both the client and server,
> > >
> > > will
> > >
> > > the producer still use the AdminClient (CreateTopics request)
> > >
> > > to
> > >
> > > create topics? and not rely on the broker auto create. I'm guessing the
> > > answer is yes but can you make it explicit.
> > >
> > > Thank you
> > >
> > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi,
> > > Just a friendly reminder to take a look at this KIP if you
> > >
> > > have
> > >
> > > the
> > >
> > > time.
> > >
> > > I was thinking about broker vs. client default precedence,
> > >
> > > and I
> > >
> > > think it
> > >
> > > makes sense to keep the broker as the default used when both
> > >
> > > client-side
> > >
> > > and broker-side defaults are configured. The idea is that
> > >
> > > there
> > >
> > > would be
> > >
> > > pretty clear documentation, and that many systems with
> > >
> > > configurations
> > >
> > > that
> > >
> > > the client could not change would likely have the auto-create
> > >
> > > default
> > >
> > > off.
> > >
> > > (In cloud for example).
> > >
> > > It also seems like in most cases, the consumer config
> > > 'allow.auto.create.topics' was created to actually prevent
> > >
> > > the
> > >
> > > creation
> > >
> > > of
> > >
> > > topics, so the loss of creation functionality will not be a
> > >
> > > big
> > >
> > > problem.
> > >
> > > I'm happy to discuss any other compatibility problems or
> > >
> > > components
> > >
> > > of
> > >
> > > this KIP.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > >
> > > jolshan@confluent.io
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I was looking at this KIP again, and there is a decision I
> > >
> > > made
> > >
> > > that I
> > >
> > > think is worth discussing.
> > >
> > > In the case where both the broker and producer's
> > > 'auto.create.topics.enable' are set to true, we have to
> > >
> > > choose
> > >
> > > either
> > >
> > > the
> > >
> > > broker configs or the producer configs for the replication
> > > factor/partitions.
> > >
> > > Currently, the decision is to have the broker defaults take
> > >
> > > precedence.
> > >
> > > (It is easier to do this in the implementation.) It also
> > >
> > > makes
> > >
> > > some
> > >
> > > sense
> > >
> > > for this behavior to take precedence since this behavior
> > >
> > > already
> > >
> > > occurs as
> > >
> > > the default.
> > >
> > > However, I was wondering if it would be odd for those who
> > >
> > > can
> > >
> > > only
> > >
> > > see
> > >
> > > the
> > >
> > > producer side to set configs for replication
> > >
> > > factor/partitions
> > >
> > > and
> > >
> > > see
> > >
> > > different results. Currently the documentation for the
> > >
> > > config
> > >
> > > states
> > >
> > > that
> > >
> > > the config values are only used when the broker config is
> > >
> > > not
> > >
> > > enabled,
> > >
> > > but
> > >
> > > this might not always be clear to the user. Changing the
> > >
> > > code
> > >
> > > to
> > >
> > > have
> > >
> > > the
> > >
> > > producer's configurations take precedence is possible, but
> > >
> > > I
> > >
> > > was
> > >
> > > wondering
> > >
> > > what everyone thought.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Just a quick update--
> > >
> > > It seems that enabling both the broker and producer
> > >
> > > configs
> > >
> > > works
> > >
> > > fine,
> > >
> > > except that the broker configurations for partitions,
> > >
> > > replication
> > >
> > > factor
> > >
> > > take precedence.
> > > I don't know if that is something we would want to
> > >
> > > change, but
> > >
> > > I'll be
> > >
> > > updating the KIP for now to reflect this. Perhaps we would
> > >
> > > want to
> > >
> > > add more
> > >
> > > to the documentation of the the producer configs to
> > >
> > > clarify.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for looking at the KIP. I can definitely add to
> > >
> > > the
> > >
> > > title
> > >
> > > to
> > >
> > > make
> > >
> > > it more clear.
> > >
> > > It makes sense that both configurations could be turned
> > >
> > > on
> > >
> > > since
> > >
> > > there
> > >
> > > are many cases where the user can not control the
> > >
> > > server-side
> > >
> > > configurations. I was a little concerned about how both
> > >
> > > interacting
> > >
> > > would
> > >
> > > work out -- if there would be to many requests for new
> > >
> > > topics,
> > >
> > > for
> > >
> > > example.
> > >
> > > But it since it does make sense to allow both
> > >
> > > configurations
> > >
> > > enabled, I
> > >
> > > will test out some scenarios and I'll change the KIP to
> > >
> > > support
> > >
> > > this.
> > >
> > > I also agree with documentation about distinguishing the
> > >
> > > differences. I
> > >
> > > was having some trouble with the wording but I like the
> > >
> > > phrases
> > >
> > > "server-side" and "client-side." That's a good
> > >
> > > distinction I
> > >
> > > can
> > >
> > > use
> > >
> > > when
> > >
> > > describing.
> > >
> > > I'll try to update the KIP soon keeping everyone's input
> > >
> > > in
> > >
> > > mind.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > >
> > > cmccabe@apache.org
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP. This seems like a good step towards
> > >
> > > removing
> > >
> > > server-side topic auto-creation.
> > >
> > > We should add included "client-side" to the title of
> > >
> > > the KIP
> > >
> > > somewhere,
> > >
> > > to make it clear that we're talking about client-side
> > >
> > > auto
> > >
> > > creation.
> > >
> > > The KIP says:
> > >
> > > In order to automatically create topics with the
> > >
> > > producer, the
> > >
> > > producer's
> > >
> > > auto.create.topics.enable config must be set to true
> > >
> > > and
> > >
> > > the
> > >
> > > broker
> > >
> > > config should be set to false
> > >
> > > From a user's point of view, this seems
> > >
> > > counter-intuitive.
> > >
> > > In
> > >
> > > order to
> > >
> > > auto-create topics the broker's
> > >
> > > auto.create.topics.enable
> > >
> > > config
> > >
> > > should be
> > >
> > > set to false? It seems like the server-side
> > >
> > > auto-create is
> > >
> > > unrelated to
> > >
> > > the client-side auto-create. We could have both turned
> > >
> > > on
> > >
> > > (and
> > >
> > > I'm
> > >
> > > sure
> > >
> > > that in the real world, people will try this
> > >
> > > configuration...)
> > >
> > > There's no
> > >
> > > reason not to support this, I think.
> > >
> > > We should add some documentation explaining the
> > >
> > > difference
> > >
> > > between
> > >
> > > server-side and client-side auto-creation. Without
> > >
> > > documentation,
> > >
> > > an admin
> > >
> > > might think that they had disabled all forms of
> > >
> > > auto-creation by
> > >
> > > setting
> > >
> > > the -side setting to false-- but this is not the case,
> > >
> > > of
> > >
> > > course.
> > >
> > > best,
> > > Colin
> > >
> > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > >
> > > Hi Dhruvil,
> > >
> > > Thanks for reading the KIP!
> > > That was the general idea for deprecation. We would
> > >
> > > log a
> > >
> > > warning
> > >
> > > when the
> > >
> > > config is enabled on the broker.
> > > I also believe that there would be a change to
> > >
> > > documentation.
> > >
> > > If there is anything else that should be done, please
> > >
> > > let
> > >
> > > me
> > >
> > > know!
> > >
> > > Justine
> > >
> > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > >
> > > dhruvil@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP, this is great!
> > >
> > > Could you add some more information about what
> > >
> > > deprecating
> > >
> > > the
> > >
> > > broker
> > >
> > > configuration means? Would we log a warning in the
> > >
> > > logs
> > >
> > > when
> > >
> > > auto
> > >
> > > topic
> > >
> > > creation is enabled on the broker, for example?
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > > deprecate the current system of
> > >
> > > auto-creating
> > >
> > > topics
> > >
> > > through requests to the metadata and give the
> > >
> > > producer the
> > >
> > > ability to
> > >
> > > automatically create topics instead.
> > >
> > > More information can be found here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > >
> > > Thank you,
> > > Justine Olshan
> > >
> > >
> >
>
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Justine Olshan <jo...@confluent.io>.

Hello all,

Thank you for all the feedback!

It seems that one of the main issues is how the client-side auto-creation
can act on its own and does not simply block or allow auto-creation as
configured by the broker.

I think I was a bit unclear about this, but the idea of this KIP is to
eventually replace the functionality of the broker config. We don't want to
check if the broker config is also enabled because the idea is that the
broker config would not be enabled, and only specific producers/clients
would be auto-creating topics.
Some older clients require auto-topic creation for only the topics they
need, and this KIP would make these clients compatible with brokers that
disable autocreation.

I now understand the worry about security and 'overriding' the broker's
auto.create.topic.enable configuration. However, in the case I explained
above, having the broker stop the producer would prevent the clients I
described above from being able to create topics. (Basically not allowing
the main point of creating this KIP.)
I'm not sure to go about having such a security feature. Perhaps adding
another config on the broker to prevent this that would be off by default
but could be turned on? It would complicate things more, but I'm open to
discussing.

Thank you,
Justine

On Tue, Aug 6, 2019 at 11:47 PM Colin McCabe <cm...@apache.org> wrote:

> On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> > Hi Colin,
> > "Hmm... I'm not sure I follow.  Users don't have to build their own
> > tooling, right?  They can use any of the shell scripts that we've shipped
> > in the last few releases.  For example, if any of your users run it, this
> > shell script will delete all of the topics from your non-security-enabled
> > cluster:
> >
> > ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
> 2>/dev/null
> > | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092
> --delete
> > --topic
> >
> > They will need to fill in the correct bootstrap servers list, of course,
> > not localhost.  This deletion script will work on some pretty old
> brokers,
> > even back to the 0.10 releases.  It seems a little odd to trust your
> users
> > with this power, but not trust them to avoid changing a particular
> > configuration key."
> >
> > The above will blocked by the server if we set delete.topic.enable to
> false
> > and thats exactly what I am asking for.
>
> Hi Harsha,
>
> I was wondering if someone was going to bring up that configuration :)
>
> it's an interesting complication, but globally disabling topic deletion is
> not very practical for most use-cases.
>
> In any case, there are plenty of other bad things that users with full
> permissions can do that aren't blocked by any server configuration.  For
> example, they can delete every record in every topic.  I can write a script
> for that too, and there's no server configuration you can set to disable
> it.  Or I could simply create hundreds of thousands of topics, until
> cluster performance becomes unacceptable (this will be even more of a
> problem if someone configured delete.topic.enable as false).  Or publish
> bad data to every topic, etc. etc.
>
> The point I'm trying to make here is that you can't rely on these kind of
> server-side configurations for security.  At most, they're a way to set up
> certain very simple policies.  But the policies are so simple that they're
> hardly ever useful any more.
>
> For example, if the problem you want to solve is that you want a user to
> only be able to create 50 topics and not delete anyone else's topics, you
> can solve that with a CreateTopicsPolicy that limits the number of topics,
> and some ACLs.  There's no combination of auto.create.topics.enable and
> delete.topic.enable that will help here.
>
> >
> > "The downside is that if we wanted to check a server side configuration
> > before sending the create topics request, the code would be more complex.
> > The behavior would also not be consistent with how topic auto-creation is
> > handled in Kafka Streams."
> >
> > I am not sure why you need to check server side configuration before
> > sending create topics request. A user enables producer side config to
> > create topics.
> > Producer sends a request to the broker and if the broker has
> > auto.topic.create.enable to true (default) it will allow creation of
> > topics. If it set to false it returns error back to the client.
>
> auto.topic.create.enable has never affected CreateTopicsRequest.  If you
> submit a CreateTopicsRequest and you are authorized, a topic will be
> created, regardless of what the value of auto.topic.create.enable is.  This
> behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?)
>
> > I don't see how this behavior will be different in Kafka streams. By
> > default server allows the topic creation and with this KIP, It will only
> > allow creation of topic when both producer and server side are turned on.
> > Its exactly the same behavior in KIP-361.
>
> I suggest running a streams job as a test.  It creates the topics it needs
> using AdminClient.  The value of auto.topic.create.enable is not relevant.
> Whether it is true or false, the topics will still be created.
>
> >
> > "In general, it would be nice if we could keep the security and access
> > control model simple and not introduce a lot of special cases and
> > exceptions.  Kafka has basically converged on using ACLs and
> > CreateTopicPolicy classes to control who can create topics and where.
> > Adding more knobs seems like a step backwards, especially when the
> > proposed knobs don't work consistently across components, and don't
> provide true
> > security." This is not about access control at all. Shipping sane
> defaults should
> > be prioritized.
>
> I don't think the default is really the question here.  I think everyone
> agrees that the default for client-side auto-topic creation should be off.
>
> The scenarios that you're describing (such as dealing with a poorly
> configured client that tries to create topics it should not) do seem to be
> about access control.
>
> > We keep talking about CreateTopicPolicy and yet we don't have default
> > one and asking every user of Kafka implement their own doesn't make
> sense
> > here.
>
> The point of CreateTopicPolicy is exactly that you can implement your own,
> though.  It's a way of customizing what the policy is in your specific
> cluster.
>
> I agree that most people don't want to write a custom policy.  But most of
> those people are satisfied with just setting up ACLs to set up simple
> policies like this user can create topics, that user can't, etc.  That's
> why I suggested adding support for ACLs to insecure clusters might be
> helpful.
>
> Also, just as a side note, wouldn't it would be impossible for us to
> specify a default CreateTopicsPolicy that did anything at this point anyway
> without breaking backwards compatibility?  Maybe I'm misunderstanding the
> proposal.
>
> > I am asking about exact behavior that we shipped for consumer side
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> >
> > I still didn't get any response why this behavior shouldn't be exactly
> like
> > Kafka consumer and why do we want to have producer to overrider server
> side
> > config and while not allowing consumer to do so.
> > We are not even allowing the same contract and this will cause more
> > confusion from the users standpoint.
>
> Hmm.  The consumer already has a way to override the server side
> configuration.  See KIP-361: Add Consumer Configuration to Disable Auto
> Topic Creation.  This lets the consumer skip auto-creating topics, even if
> auto-creation is enabled on the broker.
>
> To be fair, the KIP should probably discuss why we don't want to implement
> client-side topic creation in the consumer in "rejected alternatives."
> Maybe Justine can add more context here in the KIP.
>
> The last time we talked about this, the reasoning was that we wanted to
> eventually get rid of consumer-side auto-topic creation entirely, and so it
> wasn't worth implementing an improved way of doing it.  Producer side auto
> topic-creation, in contrast, will probably stick around, although we'd like
> to deprecate and remove the problematic server-side mechanism over time.
>
> best,
> Colin
>
> >
> >
> > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > > > Not sure how the AdminClient applies here, It is an external API and
> > > > is not part of KafkaProducer so any user who updates to latest
> version of
> > > > Kafka with this feature get to create the topics.
> > > > They have to build a tooling around AdminClient allow themselves to
> > > create
> > > > topics.
> > >
> > > Hi Harsha,
> > >
> > > Hmm... I'm not sure I follow.  Users don't have to build their own
> > > tooling, right?  They can use any of the shell scripts that we've
> shipped
> > > in the last few releases.  For example, if any of your users run it,
> this
> > > shell script will delete all of the topics from your
> non-security-enabled
> > > cluster:
> > >
> > > ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
> 2>/dev/null
> > > | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092
> --delete
> > > --topic
> > >
> > > They will need to fill in the correct bootstrap servers list, of
> course,
> > > not localhost.  This deletion script will work on some pretty old
> brokers,
> > > even back to the 0.10 releases.  It seems a little odd to trust your
> users
> > > with this power, but not trust them to avoid changing a particular
> > > configuration key.
> > >
> > > > There is no behavior in Kafka producer that allowed users to
> > > > delete the topics or delete the records. So citing them as an
> > > > example doesn't makes sense in this context.
> > >
> > > I think Kafka Streams is relevant here.  After all, it's software that
> we
> > > ship as part of the official Kafka release.  And it auto-creates topics
> > > even when auto.create.topics.enable is set to false on the broker.
> > >
> > > I think that auto.create.topics.enable was never intended as a security
> > > setting to control access.  It was intended as a way to disable the
> > > broker-side auto-create feature specifically, because there were some
> > > problems with that specific feature.  Broker-side auto-creation is
> > > frustrating because it's on by default, and because it can auto-create
> > > topics even if the producer or consumer didn't explicitly ask for that.
> > > Neither of those problems applies to this KIP: producers have to
> > > specifically opt in, and it won't be on by default.  Basically, we
> think
> > > that client-side auto-creation is actually a lot better-- hence this
> KIP :)
> > >
> > > > But there is
> > > > a functionality which allowed creation of topics if they don't exist
> in
> > > the
> > > > cluster and this behavior could be controlled by a config on the
> server
> > > > side. Now with this KIP we are allowing producer to make that
> decision
> > > > without any gateway on the server via configs. Any user who is not
> aware
> > > of
> > > > such changes
> > > > can accidentally create these topics and we are essentially removing
> a
> > > > config that exists in brokers today to block this accidental
> creation and
> > > > allowing clients to take control.
> > >
> > > Again, I hope I'm not misinterpreting, but I don't see how this can be
> > > turned on accidentally.  The user would have to specifically turn this
> on
> > > in the producer by setting the configuration key.
> > >
> > > >       I still didn't get any positives of not having server side
> configs?
> > > > if you want to turn them on and allow any client to create topics
> set the
> > > > default to true
> > > > and allow users who doesn't want to have this feature let them turn
> them
> > > > off. It will be the exact behavior as it is today, as far as
> producer is
> > > > concerned. I am not
> > > > understanding why not having server side configs to gateways such a
> hard
> > > > requirement and this behavior exists today. As far I am concerned
> this
> > > > breaks the backward compatibility.
> > >
> > > The downside is that if we wanted to check a server side configuration
> > > before sending the create topics request, the code would be more
> complex.
> > > The behavior would also not be consistent with how topic auto-creation
> is
> > > handled in Kafka Streams.
> > >
> > > In general, it would be nice if we could keep the security and access
> > > control model simple and not introduce a lot of special cases and
> > > exceptions.  Kafka has basically converged on using ACLs and
> > > CreateTopicPolicy classes to control who can create topics and where.
> > > Adding more knobs seems like a step backwards, especially when the
> proposed
> > > knobs don't work consistently across components, and don't provide true
> > > security.
> > >
> > > Maybe we should support an insecure mode where there are still users
> and
> > > ACLs.  That would help people who don't want to set up Kerberos, etc.
> but
> > > still want to protect against misconfigured clients or accidents.
> Hadoop
> > > has something like this, and I think it was useful.  NFS also supported
> > > (supports?) a mode where you just pass whatever user ID you want and
> the
> > > system believes you.  These things clearly don't protect against
> malicious
> > > users, but they can help set up policies when needed.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > Hi Harsha,
> > > > >
> > > > > Thanks for taking a look.
> > > > >
> > > > > I'm not sure I follow the discussion about AdminClient.
> > > KafkaAdminClient
> > > > > has been around for about 2 years at this point as a public class.
> > > There
> > > > > are many programs that use it to automatically create topics.
> Kafka
> > > > > Streams does this, for example.  If any of your users run Kafka
> > > Streams,
> > > > > they will be auto-creating topics, regardless of what setting you
> use
> > > for
> > > > > auto.create.topics.enable.
> > > > >
> > > > > A big part of the annoyance of auto-topic creation right now is
> that
> > > it is
> > > > > on by default.  The new configuration proposed by KIP-487 wouldn't
> be.
> > > > > Users would have to explicitly opt in to the new behavior of
> > > client-side
> > > > > topic creation.  If you run without security, you're already
> putting a
> > > huge
> > > > > amount of trust in your users.  For example, you trust them not to
> > > delete
> > > > > records with the kafka-delete-records.sh command, or delete topics
> with
> > > > > kafka-topics.sh.  Trusting them not to set a certain config value
> seems
> > > > > minor in comparison, right?
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > > > > > Hi,
> > > > > >     Even with policies one needs to implement that, so for every
> > > user who
> > > > > > doesn't want a producer to create topics or have limits around
> > > > > partitions &
> > > > > > replication factor they have to implement a policy.
> > > > > >       The KIP is changing the behavior , it might not be
> introducing
> > > the
> > > > > > new functionality but it will enable producers to override the
> create
> > > > > topic
> > > > > > config settings on the broker. What I am asking for to provide a
> > > config
> > > > > > that will disable auto creation of topics and if its enabled set
> some
> > > > > sane
> > > > > > defaults so that clients can create a topic with in those
> limits. I
> > > don't
> > > > > > see how this not related to this KIP.
> > > > > >      If the server config options as I suggested doesn't interest
> > > you at
> > > > > > least have a default CreateTopicPolices in place.
> > > > > >        To give an example, In our environment we disable the
> > > > > > auto.create.topic.enable and force users to go through a
> centralized
> > > > > > service as we want capture more details about the topic creation
> and
> > > > > > requirements. With this KIP, a producer can create a topic with
> no
> > > > > bounds.
> > > > > >  Another example max.message.size we define that at cluster level
> > > and one
> > > > > > can override max.messsage.size at topic level, any
> misconfiguration
> > > at
> > > > > this
> > > > > > will cause service degradation.  Its not always about the rogue
> > > clients,
> > > > > > users can easily misconfigure and can cause an outage.
> > > > > > Again we can talk about CreateTopicPolicy but without having a
> > > default
> > > > > > implementation and asking everyone to implement their own while
> > > changing
> > > > > > the behavior in producer  doesn't make sense to me.
> > > > > >
> > > > > > Thanks,
> > > > > > Harsha
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > > >
> > > > > > > Hi Harsha,
> > > > > > >
> > > > > > > I mentioned policies and the authorizer. For example, with
> > > > > > > CreateTopicPolicy, you can implement the limits you describe.
> If
> > > you
> > > > > have
> > > > > > > ideas of how that should be improved, please submit a KIP. My
> > > point is
> > > > > that
> > > > > > > this KIP is not introducing any new functionality with regards
> to
> > > what
> > > > > > > rogue clients can do. It's using the existing protocol that is
> > > already
> > > > > > > exposed via the AdminClient. So, I don't think we need to
> address
> > > it in
> > > > > > > this KIP. Does that make sense?
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <
> > > kafka@harsha.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > Ismael,
> > > > > > > Sure AdminClient can do that and we should've shipped a config
> or
> > > use
> > > > > the
> > > > > > > existing one to block that. Not all users are yet to upgrade to
> > > > > AdminClient
> > > > > > > and start using that to cause issues yet. In shared
> environment we
> > > > > should
> > > > > > > allow server to set sane defaults and not allow every client
> to go
> > > > > ahead
> > > > > > > create random no.of topic/partitions and replication factor.
> Even
> > > if
> > > > > the
> > > > > > > users want to allow topic creation proposed in the KIP , it
> makes
> > > > > sense to
> > > > > > > have some guards against the no.of partitions and replication
> > > factor.
> > > > > > > Authorizer is not always an answer to block requests and having
> > > users
> > > > > set
> > > > > > > server side configs to protect a multi-tenant environment is
> > > required.
> > > > > In a
> > > > > > > non-secure environment Authorizer is a blunt instrument either
> you
> > > end
> > > > > up
> > > > > > > blocking everyone or allowing everyone.
> > > > > > > I am asking to have server side that allow clients to create
> > > topics or
> > > > > not
> > > > > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > > > > replication-factor.
> > > > > > >
> > > > > > > -Harsha
> > > > > > >
> > > > > > > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> > > > > > >
> > > > > > > Harsha,
> > > > > > >
> > > > > > > Rogue clients can use the admin client to create topics and
> > > partitions.
> > > > > > > ACLs and policies can help in that case as well as this one.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <
> kafka@harsha.io>
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi Justine,
> > > > > > > Thanks for the KIP.
> > > > > > > "When server-side auto-creation is disabled, client-side
> > > auto-creation
> > > > > > > will try to use client-side configurations"
> > > > > > > If I understand correctly, this KIP is removing any server-side
> > > > > blocking
> > > > > > > client auto creation of topic?
> > > > > > > if so this will present potential issue of rogue client
> creating
> > > ton of
> > > > > > > topic-partitions and potentially bringing down the service for
> > > everyone
> > > > > > >
> > > > > > > or
> > > > > > >
> > > > > > > degrade the service itself.
> > > > > > > By reading the KIP its not clear to me that there is a clear
> way to
> > > > > block
> > > > > > > auto creation topics of all together from clients by server
> side
> > > > > config.
> > > > > > > Server side configs of default topic, partitions should take
> higher
> > > > > > > precedence and client shouldn't be able to create a topic with
> > > higher
> > > > > > >
> > > > > > > no.of
> > > > > > >
> > > > > > > partitions, replication than what server config specifies.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Harsha
> > > > > > >
> > > > > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <
> > > jolshan@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > > I made some changes to the KIP. Hopefully this configuration
> change
> > > > > > >
> > > > > > > will
> > > > > > >
> > > > > > > make things a little clearer.
> > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > > > > >
> > > > > > > Please let me know if you have any feedback or questions!
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <
> cmccabe@apache.org>
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > I think you bring up a good point. It would be better if we
> didn't
> > > ever
> > > > > > > have to set up client-side configuration for this feature, and
> > > KIP-464
> > > > > > > would let us skip this entirely.
> > > > > > >
> > > > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > > I agree that KIP-464 works on newer brokers, but I was a bit
> > > worried
> > > > > > >
> > > > > > > how
> > > > > > >
> > > > > > > things would play out on older brokers that* do not *have KIP
> 464
> > > > > > >
> > > > > > > included.
> > > > > > >
> > > > > > > Is it enough to throw an error in this case when producer
> configs
> > > are
> > > > > > >
> > > > > > > not
> > > > > > >
> > > > > > > specified?
> > > > > > >
> > > > > > > I think the right thing to do would be to log an error message
> in
> > > the
> > > > > > > client. We will need to have that capability in any case, to
> cover
> > > > > > > scenarios like the client trying to auto-create a topic that
> they
> > > don't
> > > > > > > have permission to create. Or a client trying to create a
> topic on
> > > a
> > > > > > >
> > > > > > > broker
> > > > > > >
> > > > > > > so old that CreateTopicsRequest is not supported.
> > > > > > >
> > > > > > > The big downside to relying on KIP-464 is that it is a very
> recent
> > > > > > >
> > > > > > > feature
> > > > > > >
> > > > > > > -- so recent that it hasn't even made its way to any official
> > > Apache
> > > > > > > release. It's scheduled for the upcoming 2.4 release in a few
> > > months.
> > > > > > >
> > > > > > > So if you view this KIP as a step towards removing broker-side
> > > > > > > auto-create, you might want to support older brokers just to
> > > accelerate
> > > > > > > adoption, and hasten the day when we can finally flip
> broker-side
> > > > > > > auto-create to off (or even remove it entirely).
> > > > > > >
> > > > > > > I have to agree, though, that having client-side
> configurations for
> > > > > > >
> > > > > > > number
> > > > > > >
> > > > > > > of partitions and replication factor is messy. Maybe it would
> be
> > > worth
> > > > > > >
> > > > > > > it
> > > > > > >
> > > > > > > to restrict support to post-KIP-464 brokers, if we could avoid
> > > creating
> > > > > > > more configs.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> > > > > > >
> > > > > > > mickael.maison@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > We can rely on KIP-464 which allows to omit the partition
> count or
> > > > > > > replication factor when creating a topic. In that case, the
> broker
> > > > > > >
> > > > > > > defaults
> > > > > > >
> > > > > > > are used.
> > > > > > >
> > > > > > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <
> > > jolshan@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > Michael,
> > > > > > > That makes sense to me!
> > > > > > > To clarify, in the current state of the KIP, the producer does
> not
> > > > > > >
> > > > > > > rely
> > > > > > >
> > > > > > > on
> > > > > > >
> > > > > > > the broker to autocreate--if the broker's config is disabled,
> then
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > producer can autocreate on its own with a create topic request
> (the
> > > > > > >
> > > > > > > same
> > > > > > >
> > > > > > > type of request the admin client uses).
> > > > > > > However, if both configs are enabled, the broker will
> autocreate
> > > > > > >
> > > > > > > through
> > > > > > >
> > > > > > > a
> > > > > > >
> > > > > > > metadata request before the producer gets a chance. Of course,
> the
> > > way
> > > > > > >
> > > > > > > to
> > > > > > >
> > > > > > > avoid this, is to do as you suggested, and set
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > "allow_auto_topic_creation" field to false.
> > > > > > >
> > > > > > > I think the only thing we need to be careful with in this
> setup is
> > > > > > >
> > > > > > > without
> > > > > > >
> > > > > > > KIP 464, we can not use broker defaults for this topic. A user
> > > needs
> > > > > > >
> > > > > > > to
> > > > > > >
> > > > > > > specify the number of partition and replication factor in the
> > > config.
> > > > > > >
> > > > > > > An
> > > > > > >
> > > > > > > alternative to this is to have coded defaults for when these
> > > > > > >
> > > > > > > configs
> > > > > > >
> > > > > > > are
> > > > > > >
> > > > > > > unspecified, but it is not immediately apparent what these
> defaults
> > > > > > >
> > > > > > > should
> > > > > > >
> > > > > > > be.
> > > > > > >
> > > > > > > Thanks again for reading my KIP,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > > > > > >
> > > > > > > mickael.maison@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Thanks for the response!
> > > > > > > In my opinion, it would be better if the producer did not rely
> at
> > > > > > >
> > > > > > > all
> > > > > > >
> > > > > > > on the broker auto create feature as this is what we're aiming
> to
> > > > > > > deprecate. When requesting metadata we can set the
> > > > > > > "allow_auto_topic_creation" field to false to avoid the broker
> auto
> > > > > > > creation. Then if the topic is not existing, send a
> > > CreateTopicRequest.
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > > > > > >
> > > > > > > jolshan@confluent.io>
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Currently the way it is implemented, the broker auto-creation
> > > > > > >
> > > > > > > configuration
> > > > > > >
> > > > > > > takes precedence. The producer will not use the CreateTopics
> > > > > > >
> > > > > > > request.
> > > > > > >
> > > > > > > (Technically it can--but the topic will already be created
> > > > > > >
> > > > > > > through
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > broker, so it will never try to create the topic.) It is
> possible
> > > to
> > > > > > > change this however, and I'd be happy to
> > > > > > >
> > > > > > > discuss
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > benefits of this alternative.
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > > > > > >
> > > > > > > mickael.maison@gmail.com>
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Thanks for the KIP!
> > > > > > >
> > > > > > > In case auto creation is enabled on both the client and server,
> > > > > > >
> > > > > > > will
> > > > > > >
> > > > > > > the producer still use the AdminClient (CreateTopics request)
> > > > > > >
> > > > > > > to
> > > > > > >
> > > > > > > create topics? and not rely on the broker auto create. I'm
> > > guessing the
> > > > > > > answer is yes but can you make it explicit.
> > > > > > >
> > > > > > > Thank you
> > > > > > >
> > > > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > > > > > >
> > > > > > > jolshan@confluent.io>
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > > Just a friendly reminder to take a look at this KIP if you
> > > > > > >
> > > > > > > have
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > time.
> > > > > > >
> > > > > > > I was thinking about broker vs. client default precedence,
> > > > > > >
> > > > > > > and I
> > > > > > >
> > > > > > > think it
> > > > > > >
> > > > > > > makes sense to keep the broker as the default used when both
> > > > > > >
> > > > > > > client-side
> > > > > > >
> > > > > > > and broker-side defaults are configured. The idea is that
> > > > > > >
> > > > > > > there
> > > > > > >
> > > > > > > would be
> > > > > > >
> > > > > > > pretty clear documentation, and that many systems with
> > > > > > >
> > > > > > > configurations
> > > > > > >
> > > > > > > that
> > > > > > >
> > > > > > > the client could not change would likely have the auto-create
> > > > > > >
> > > > > > > default
> > > > > > >
> > > > > > > off.
> > > > > > >
> > > > > > > (In cloud for example).
> > > > > > >
> > > > > > > It also seems like in most cases, the consumer config
> > > > > > > 'allow.auto.create.topics' was created to actually prevent
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > creation
> > > > > > >
> > > > > > > of
> > > > > > >
> > > > > > > topics, so the loss of creation functionality will not be a
> > > > > > >
> > > > > > > big
> > > > > > >
> > > > > > > problem.
> > > > > > >
> > > > > > > I'm happy to discuss any other compatibility problems or
> > > > > > >
> > > > > > > components
> > > > > > >
> > > > > > > of
> > > > > > >
> > > > > > > this KIP.
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > > > > > >
> > > > > > > jolshan@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hello all,
> > > > > > >
> > > > > > > I was looking at this KIP again, and there is a decision I
> > > > > > >
> > > > > > > made
> > > > > > >
> > > > > > > that I
> > > > > > >
> > > > > > > think is worth discussing.
> > > > > > >
> > > > > > > In the case where both the broker and producer's
> > > > > > > 'auto.create.topics.enable' are set to true, we have to
> > > > > > >
> > > > > > > choose
> > > > > > >
> > > > > > > either
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > broker configs or the producer configs for the replication
> > > > > > > factor/partitions.
> > > > > > >
> > > > > > > Currently, the decision is to have the broker defaults take
> > > > > > >
> > > > > > > precedence.
> > > > > > >
> > > > > > > (It is easier to do this in the implementation.) It also
> > > > > > >
> > > > > > > makes
> > > > > > >
> > > > > > > some
> > > > > > >
> > > > > > > sense
> > > > > > >
> > > > > > > for this behavior to take precedence since this behavior
> > > > > > >
> > > > > > > already
> > > > > > >
> > > > > > > occurs as
> > > > > > >
> > > > > > > the default.
> > > > > > >
> > > > > > > However, I was wondering if it would be odd for those who
> > > > > > >
> > > > > > > can
> > > > > > >
> > > > > > > only
> > > > > > >
> > > > > > > see
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > producer side to set configs for replication
> > > > > > >
> > > > > > > factor/partitions
> > > > > > >
> > > > > > > and
> > > > > > >
> > > > > > > see
> > > > > > >
> > > > > > > different results. Currently the documentation for the
> > > > > > >
> > > > > > > config
> > > > > > >
> > > > > > > states
> > > > > > >
> > > > > > > that
> > > > > > >
> > > > > > > the config values are only used when the broker config is
> > > > > > >
> > > > > > > not
> > > > > > >
> > > > > > > enabled,
> > > > > > >
> > > > > > > but
> > > > > > >
> > > > > > > this might not always be clear to the user. Changing the
> > > > > > >
> > > > > > > code
> > > > > > >
> > > > > > > to
> > > > > > >
> > > > > > > have
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > producer's configurations take precedence is possible, but
> > > > > > >
> > > > > > > I
> > > > > > >
> > > > > > > was
> > > > > > >
> > > > > > > wondering
> > > > > > >
> > > > > > > what everyone thought.
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > > > > > >
> > > > > > > jolshan@confluent.io>
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Just a quick update--
> > > > > > >
> > > > > > > It seems that enabling both the broker and producer
> > > > > > >
> > > > > > > configs
> > > > > > >
> > > > > > > works
> > > > > > >
> > > > > > > fine,
> > > > > > >
> > > > > > > except that the broker configurations for partitions,
> > > > > > >
> > > > > > > replication
> > > > > > >
> > > > > > > factor
> > > > > > >
> > > > > > > take precedence.
> > > > > > > I don't know if that is something we would want to
> > > > > > >
> > > > > > > change, but
> > > > > > >
> > > > > > > I'll be
> > > > > > >
> > > > > > > updating the KIP for now to reflect this. Perhaps we would
> > > > > > >
> > > > > > > want to
> > > > > > >
> > > > > > > add more
> > > > > > >
> > > > > > > to the documentation of the the producer configs to
> > > > > > >
> > > > > > > clarify.
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > > > > > >
> > > > > > > jolshan@confluent.io>
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi Colin,
> > > > > > >
> > > > > > > Thanks for looking at the KIP. I can definitely add to
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > title
> > > > > > >
> > > > > > > to
> > > > > > >
> > > > > > > make
> > > > > > >
> > > > > > > it more clear.
> > > > > > >
> > > > > > > It makes sense that both configurations could be turned
> > > > > > >
> > > > > > > on
> > > > > > >
> > > > > > > since
> > > > > > >
> > > > > > > there
> > > > > > >
> > > > > > > are many cases where the user can not control the
> > > > > > >
> > > > > > > server-side
> > > > > > >
> > > > > > > configurations. I was a little concerned about how both
> > > > > > >
> > > > > > > interacting
> > > > > > >
> > > > > > > would
> > > > > > >
> > > > > > > work out -- if there would be to many requests for new
> > > > > > >
> > > > > > > topics,
> > > > > > >
> > > > > > > for
> > > > > > >
> > > > > > > example.
> > > > > > >
> > > > > > > But it since it does make sense to allow both
> > > > > > >
> > > > > > > configurations
> > > > > > >
> > > > > > > enabled, I
> > > > > > >
> > > > > > > will test out some scenarios and I'll change the KIP to
> > > > > > >
> > > > > > > support
> > > > > > >
> > > > > > > this.
> > > > > > >
> > > > > > > I also agree with documentation about distinguishing the
> > > > > > >
> > > > > > > differences. I
> > > > > > >
> > > > > > > was having some trouble with the wording but I like the
> > > > > > >
> > > > > > > phrases
> > > > > > >
> > > > > > > "server-side" and "client-side." That's a good
> > > > > > >
> > > > > > > distinction I
> > > > > > >
> > > > > > > can
> > > > > > >
> > > > > > > use
> > > > > > >
> > > > > > > when
> > > > > > >
> > > > > > > describing.
> > > > > > >
> > > > > > > I'll try to update the KIP soon keeping everyone's input
> > > > > > >
> > > > > > > in
> > > > > > >
> > > > > > > mind.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > > > > > >
> > > > > > > cmccabe@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Thanks for the KIP. This seems like a good step towards
> > > > > > >
> > > > > > > removing
> > > > > > >
> > > > > > > server-side topic auto-creation.
> > > > > > >
> > > > > > > We should add included "client-side" to the title of
> > > > > > >
> > > > > > > the KIP
> > > > > > >
> > > > > > > somewhere,
> > > > > > >
> > > > > > > to make it clear that we're talking about client-side
> > > > > > >
> > > > > > > auto
> > > > > > >
> > > > > > > creation.
> > > > > > >
> > > > > > > The KIP says:
> > > > > > >
> > > > > > > In order to automatically create topics with the
> > > > > > >
> > > > > > > producer, the
> > > > > > >
> > > > > > > producer's
> > > > > > >
> > > > > > > auto.create.topics.enable config must be set to true
> > > > > > >
> > > > > > > and
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > broker
> > > > > > >
> > > > > > > config should be set to false
> > > > > > >
> > > > > > > From a user's point of view, this seems
> > > > > > >
> > > > > > > counter-intuitive.
> > > > > > >
> > > > > > > In
> > > > > > >
> > > > > > > order to
> > > > > > >
> > > > > > > auto-create topics the broker's
> > > > > > >
> > > > > > > auto.create.topics.enable
> > > > > > >
> > > > > > > config
> > > > > > >
> > > > > > > should be
> > > > > > >
> > > > > > > set to false? It seems like the server-side
> > > > > > >
> > > > > > > auto-create is
> > > > > > >
> > > > > > > unrelated to
> > > > > > >
> > > > > > > the client-side auto-create. We could have both turned
> > > > > > >
> > > > > > > on
> > > > > > >
> > > > > > > (and
> > > > > > >
> > > > > > > I'm
> > > > > > >
> > > > > > > sure
> > > > > > >
> > > > > > > that in the real world, people will try this
> > > > > > >
> > > > > > > configuration...)
> > > > > > >
> > > > > > > There's no
> > > > > > >
> > > > > > > reason not to support this, I think.
> > > > > > >
> > > > > > > We should add some documentation explaining the
> > > > > > >
> > > > > > > difference
> > > > > > >
> > > > > > > between
> > > > > > >
> > > > > > > server-side and client-side auto-creation. Without
> > > > > > >
> > > > > > > documentation,
> > > > > > >
> > > > > > > an admin
> > > > > > >
> > > > > > > might think that they had disabled all forms of
> > > > > > >
> > > > > > > auto-creation by
> > > > > > >
> > > > > > > setting
> > > > > > >
> > > > > > > the -side setting to false-- but this is not the case,
> > > > > > >
> > > > > > > of
> > > > > > >
> > > > > > > course.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > > > > >
> > > > > > > Hi Dhruvil,
> > > > > > >
> > > > > > > Thanks for reading the KIP!
> > > > > > > That was the general idea for deprecation. We would
> > > > > > >
> > > > > > > log a
> > > > > > >
> > > > > > > warning
> > > > > > >
> > > > > > > when the
> > > > > > >
> > > > > > > config is enabled on the broker.
> > > > > > > I also believe that there would be a change to
> > > > > > >
> > > > > > > documentation.
> > > > > > >
> > > > > > > If there is anything else that should be done, please
> > > > > > >
> > > > > > > let
> > > > > > >
> > > > > > > me
> > > > > > >
> > > > > > > know!
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > > > > > >
> > > > > > > dhruvil@confluent.io>
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Thanks for the KIP, this is great!
> > > > > > >
> > > > > > > Could you add some more information about what
> > > > > > >
> > > > > > > deprecating
> > > > > > >
> > > > > > > the
> > > > > > >
> > > > > > > broker
> > > > > > >
> > > > > > > configuration means? Would we log a warning in the
> > > > > > >
> > > > > > > logs
> > > > > > >
> > > > > > > when
> > > > > > >
> > > > > > > auto
> > > > > > >
> > > > > > > topic
> > > > > > >
> > > > > > > creation is enabled on the broker, for example?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dhruvil
> > > > > > >
> > > > > > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > > > > > >
> > > > > > > jolshan@confluent.io>
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hello all,
> > > > > > >
> > > > > > > I'd like to start a discussion thread for KIP-487. This KIP
> plans
> > > to
> > > > > > > deprecate the current system of
> > > > > > >
> > > > > > > auto-creating
> > > > > > >
> > > > > > > topics
> > > > > > >
> > > > > > > through requests to the metadata and give the
> > > > > > >
> > > > > > > producer the
> > > > > > >
> > > > > > > ability to
> > > > > > >
> > > > > > > automatically create topics instead.
> > > > > > >
> > > > > > > More information can be found here:
> > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Justine Olshan
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

回复: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Jiamei Xie <Ji...@arm.com>.

Hi Boyang, Justin,

" 1. Could we include a link to KIP-464 and explain its relation to KIP-487?
It's very hard to read through the proposal when readers only have a
reference number to some KIP that is not briefed. "
For KIP-464  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708722,  Its motivation is to make default `num.partitions` and `default.replication.factor` available to AdminClient APIs.

For KIP-487  https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer,  I think its motivation is to make auto-create-topic configurable from producer side.


"2. The KIP suggests, " In the producer, auto-creation of a topic will occur
through a specific request rather than through a side effect of requesting
metadata." Could we be specific such as whether we are going to introduce a
separate RPC, or just send another CreateTopicRequest?"

I think there is no need to introduce a separate RPC.

MetadataRequest already has field allowAutoTopicCreation. For current ProducerMetadata, it sets allowAutoTopicCreation to true as https://github.com/apache/kafka/blob/448e7d7f0f46db1eae14d4fe7a1d25b7af894b09/clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerMetadata.java#L59.

When broker received a metadata request and the topic doesn't exist, it will decide whether to create topic according to "allowAutoTopicCreation && config.autoCreateTopicsEnable" https://github.com/apache/kafka/blob/9a4f00f78bf37041006ae8b6432d194f603ac6cc/core/src/main/scala/kafka/server/KafkaApis.scala#L1107

So the way I implemented it is different from Justine's.
Justine's PR: https://github.com/apache/kafka/pull/7075. It was implemented it by CreateTopicsRequest

Jiamei's PR : https://github.com/apache/kafka/pull/8831. I added field " allowAutoTopicCreation" to ProducerMetadata, pass it to MetadataRequest. builder.

As boyang said, " Won't it be more natural to assume that only when
both server and client agree on turning on the switch, will a topic get
created?"
It was the same as what I thought.

-----邮件原件-----
发件人: Boyang Chen <re...@gmail.com>
发送时间: 2020年6月24日 13:12
收件人: dev <de...@kafka.apache.org>
主题: Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hey Justin and Jiamei,

I read the KIP and skimmed over the discussion. One thing I'm not fully
convinced of is why we need to deprecate the server side auto topic
creation logic, which seems orthogonal towards whether a client wants to
create the topic or not. Won't it be more natural to assume that only when
both server and client agree on turning on the switch, will a topic get
created?

Some clarifications would also be appreciated:

1. Could we include a link to KIP-464 and explain its relation to KIP-487?
It's very hard to read through the proposal when readers only have a
reference number to some KIP that is not briefed.

2. The KIP suggests, " In the producer, auto-creation of a topic will occur
through a specific request rather than through a side effect of requesting
metadata." Could we be specific such as whether we are going to introduce a
separate RPC, or just send another CreateTopicRequest?

Boyang

On Wed, Jun 17, 2020 at 8:51 AM jiamei xie <ji...@gmail.com> wrote:

> Hi all
> For
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> ,  It has not been updated for a long time. And I made some update, which
> has been pushed to https://github.com/apache/kafka/pull/8831
>
> MetadataRequest has method Builder(List<String> topics, boolean
> allowAutoTopicCreation) by which we can set whether to enable
> allowAutoTopicCreation from producer.
> By default, allowAutoTopicCreation on Producer is true. And only if when
> the allowAutoTopicCreation of Broker and Producer are true, the topic can
> be auto-created.
>
> Besides, the test cases are changed:
> There are 4 cases for brokerAutoTopicCreationEnable and
> producerAutoCreateTopicsPolicy, Check if the topic is created under these
> four cases.
>      If brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy
> are true:  assertTrue(topicCreated)
>      else : intercept[ExecutionException]
>
> Looking forward to your feedback and comments. Thanks.
>
> Best wishes
> Jiamei Xie
>
> On 2019/08/12 15:50:22, Harsha Chintalapani <ka...@harsha.io> wrote:
> > On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi all,
> > >
> > > A few points:
> > >
> > > 1. I think the way backwards compatibility is being used here is not
> > > correct. Any functionality that is only enabled if set via a config is
> > > backwards compatible. People may disagree with the functionality or the
> > > config, but it's not a backwards compatibility issue.
> > >
> >
> > We are talking about both broker and producer as a  single entity and run
> > by the same team/users. Allowing newer producer to create topics on a
> older
> > broker when auto.create.topics.enable set to false, breaks  server side
> > contract that this config offered from the beginning.  IMO, it clearly
> > isn't backward compatible. User who set auto.create.topic.enable on
> broker
> > will not be the same who will turn it on producer side .
> >
> >
> > > 2. It's an interesting question if auto topic creation via the producer
> > > should be a server driven choice or not. I can see the desire to have a
> > > server-driven default, but it seems like this is often application
> > > specific. Because the functionality is trivially available via
> AdminClient
> > > (released 2 years ago), it's not quite possible to control what
> > > applications do without the use of ACLs or policies today.
> > >
> > >
> > >
> > Producers & consumers are the majority of the clients in Kafka ecosystem.
> > Just because AdminClient shipped a while back that doesn't mean all users
> > adopting to it. To this day lot more users are aware of Producer &
> Consumer
> > APIs and running them in production compare to AdminClient.
> >
> >
> > > 3. Changing the create topics request in this way is highly
> unintuitive in
> > > my opinion and it relies on every client to pass the new field. For
> > > example, if librdkafka added auto create functionality by relying on
> their
> > > AdminClient, it would behave differently than what is proposed here.
> > > Forcing every client to implement this change when calling auto create
> from
> > > the producer specifically seems odd
> > >
> >
> > I am not sure why its unintuitive , protocols change. We add or upgrade
> the
> > existing protocols all the time.
> >
> >
> > Thanks,
> > Harsha
> >
> > .
> > >
> > > Ismael
> > >
> > > On Thu, Aug 8, 2019 at 11:51 AM Jun Rao <ju...@confluent.io> wrote:
> > >
> > > Hi, Justine,
> > >
> > > Thanks for the KIP. Overall, it seems to be a good improvement.
> > >
> > > However, I think Harsha's point seems reasonable. We had
> > > auto.create.topics.enable config on the broker to allow admins to
> disable
> > > topic creation from the producer/consumer clients before we had the
> > > security feature. The need for that config is reduced with the security
> > > feature, but may still be present since not all places have security
> > > enabled. It's true that a non-secured environment is vulnerable to some
> > > additional attacks, but producer/consumer are the most common way for a
> > > user to interact with the broker. So, keeping that config for backward
> > > compatibility could still be useful if it's not introducing too much
> effort
> > > or extra confusion.
> > >
> > >
> > > Here is a one potential alternative way that I was thinking. We add a
> new
> > > field in the CreateTopicRequest to indicate whether it's from the
> producer
> > > or not. If auto.create.topics.enable is false, CreateTopicRequest from
> the
> > > producer will be rejected. We probably don't need to introduce the new
> > > config (which seems a bit hard to explain) in the producer. Instead,
> the
> > > new producer always uses MetadataRequest with AllowAutoTopicCreation
> set to
> > > false to get the metadata and if the metadata is not present, send the
> new
> > > CreateTopicRequest
> > > (assuming the broker supports it) to try to create the topic
> > > automatically. Whether the creation is allowed or not will be
> determined by
> > > the broker. This will make the behavior backward compatible and we can
> > > still achieve the main goal of the KIP, which is not relying on
> > > MetadataRequest for topic creation. What do you think?
> > >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Aug 8, 2019 at 1:34 AM M. Manna <ma...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > >
> > > If I may, perhaps you could simplify everything by using only
> > > 'auto.create.topics.enable' as a value along with true. In other words,
> > >
> > >
> > > the
> > >
> > > public interfaces section should only have
> > >
> > > [true,auto.create.topics.enable,
> > >
> > > false].
> > >
> > > The reason for this is that auto.create.topics.enable is already known
> to
> > > users as a "Server-SIde" config. So all you are saying is
> > >
> > > a) To avoid day 1 impact, it will follow whatever
> > >
> > > auto.create.topics.enable
> > >
> > >
> > > value is set.
> > > b) False means - no client side topic creation
> > > c) True means client side topic creation.
> > >
> > >
> > > It saves creating 2 more new strings :). But not too expensive anyway.
> > >
> > > Also, when you deprecate auto.create.topics.enable - you must provide
> > > sufficient logic to ensure that things like rolling upgrade doesn't
> > > temporarily break anything. I apologise if you have already accounted
> for
> > > this, but wanted to mention since I didn't notice this on the KIP.
> > >
> > > Let me know how this sounds.
> > >
> > > Regards,
> > >
> > > On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jo...@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Harsha,
> > >
> > > I think my message may have gotten lost in all the others.
> > >
> > > Two of the goals of this KIP are to 1) allow auto-creation on specific
> > > clients when the broker default is false and 2) eventually replace the
> > > broker config.
> > >
> > > In order to accomplish these two goals, we need the producer to be able
> > >
> > > to
> > >
> > >
> > > create topics despite the broker config. (How can we replace a function
> > > when we rely on it?)
> > > I think at this point we have a fundamental disagreement in what we
> > >
> > >
> > > should
> > >
> > >
> > > allow the producer to do.
> > > In my previous message I mentioned a config that would allow for the
> > >
> > >
> > > broker
> > >
> > > to prevent producer auto-creation. (It would be disabled by default.)
> > >
> > > It
> > >
> > > would fix your issue for now, but could lead to more complications
> > >
> > > later.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz>
> > >
> > > wrote:
> > >
> > > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> > >
> > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe <
> > >
> > > cmccabe@apache.org
> > >
> > > wrote:
> > >
> > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> > >
> > >
> > > Hi Colin,
> > > "Hmm... I'm not sure I follow. Users don't have to build their own
> > > tooling, right? They can use any of the shell scripts that we've
> > >
> > >
> > > shipped
> > >
> > > in
> > >
> > > the last few releases. For example, if any of your users run it,
> > >
> > > this
> > >
> > > shell
> > >
> > > script will delete all of the topics from your non-security-enabled
> > > cluster:
> > >
> > >
> > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --delete
> > > --topic
> > >
> > >
> > > They will need to fill in the correct bootstrap servers list, of
> > >
> > > course,
> > >
> > > not localhost. This deletion script will work on some pretty old
> > >
> > > brokers,
> > >
> > > even back to the 0.10 releases. It seems a little odd to trust your
> > >
> > > users
> > >
> > > with this power, but not trust them to avoid changing a particular
> > > configuration key."
> > >
> > > The above will blocked by the server if we set delete.topic.enable
> > >
> > > to
> > >
> > > false and thats exactly what I am asking for.
> > >
> > > Hi Harsha,
> > >
> > > I was wondering if someone was going to bring up that configuration
> > >
> > > :)
> > >
> > > it's an interesting complication, but globally disabling topic
> > >
> > > deletion
> > >
> > > is
> > >
> > > not very practical for most use-cases.
> > >
> > > In any case, there are plenty of other bad things that users with
> > >
> > > full
> > >
> > > permissions can do that aren't blocked by any server configuration.
> > >
> > > For
> > >
> > > example, they can delete every record in every topic. I can write a
> > >
> > > script
> > >
> > > for that too, and there's no server configuration you can set to
> > >
> > > disable
> > >
> > > it. Or I could simply create hundreds of thousands of topics, until
> > >
> > > cluster
> > >
> > > performance becomes unacceptable (this will be even more of a
> > >
> > > problem
> > >
> > > if
> > >
> > > someone configured delete.topic.enable as false). Or publish bad
> > >
> > > data
> > >
> > > to
> > >
> > > every topic, etc. etc.
> > >
> > > The point I'm trying to make here is that you can't rely on these
> > >
> > > kind
> > >
> > > of
> > >
> > > server-side configurations for security. At most, they're a way to
> > >
> > > set
> > >
> > > up
> > >
> > > certain very simple policies. But the policies are so simple that
> > >
> > > they're
> > >
> > > hardly ever useful any more.
> > >
> > > For example, if the problem you want to solve is that you want a
> > >
> > > user
> > >
> > > to
> > >
> > > only be able to create 50 topics and not delete anyone else's
> > >
> > > topics,
> > >
> > > you
> > >
> > > can solve that with a CreateTopicsPolicy that limits the number of
> > >
> > > topics,
> > >
> > > and some ACLs. There's no combination of auto.create.topics.enable
> > >
> > > and
> > >
> > > delete.topic.enable that will help here.
> > >
> > > Hi Colin,
> > >
> > > Well you gave the example that a user can delete the
> > >
> > > topics
> > >
> > > just by running that script :).
> > >
> > > I understand there are open APIs in Kafka and can lead to rogue
> > >
> > > clients
> > >
> > > taking advantage of it without proper security in place.
> > >
> > > What I am asking so far in this thread is , this KIP is changing
> > >
> > > the
> > >
> > > producer behavior and its not backward compatible.
> > >
> > > The change is backwards compatible. The default will still be
> > >
> > > server-side
> > >
> > > topic auto-creation, just like now.
> > >
> > > You will have to specifically change the producer config to get the
> > >
> > > new
> > >
> > > behavior.
> > >
> > > I disagree. Today server can turn off the topic creation neither
> > >
> > > producer
> > >
> > > or consumer can create a topic. With this KIP , producer can create a
> > >
> > > topic
> > >
> > > by turning on client side config when server side config is turned
> > >
> > > off.
> > >
> > > We can still achieve
> > >
> > > the main goal of the KIP which is to change MetadataRequest
> > >
> > > creating
> > >
> > > topics and send a CreateTopicRequest from Producer and also keep
> > >
> > > the
> > >
> > > server
> > >
> > > side config to have precedence. This KIP originally written to
> > >
> > > have
> > >
> > > server
> > >
> > > side preference and there is not much explanation why it changed to
> > >
> > > have
> > >
> > > Producer side preference.
> > >
> > > Arguing that AdminClient can do that and so we are going to make
> > >
> > > Producer
> > >
> > > do the same doesn't make sense.
> > >
> > > "The downside is that if we wanted to check a server side
> > >
> > > configuration
> > >
> > > before sending the create topics request, the code would be more
> > >
> > > complex.
> > >
> > > The behavior would also not be consistent with how topic
> > >
> > > auto-creation
> > >
> > > is
> > >
> > > handled in Kafka Streams."
> > >
> > > I am not sure why you need to check server side configuration
> > >
> > > before
> > >
> > > sending create topics request. A user enables producer side config
> > >
> > > to
> > >
> > >
> > > create topics.
> > > Producer sends a request to the broker and if the broker has
> > > auto.topic.create.enable to true (default) it will allow creation
> > >
> > >
> > > of
> > >
> > > topics. If it set to false it returns error back to the client.
> > >
> > > auto.topic.create.enable has never affected CreateTopicsRequest. If
> > >
> > > you
> > >
> > > submit a CreateTopicsRequest and you are authorized, a topic will
> > >
> > > be
> > >
> > > created, regardless of what the value of auto.topic.create.enable
> > >
> > > is.
> > >
> > > This
> > >
> > > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I
> > >
> > > think?)
> > >
> > > I don't see how this behavior will be different in Kafka streams.
> > >
> > > By
> > >
> > > default server allows the topic creation and with this KIP, It will
> > >
> > > only
> > >
> > > allow creation of topic when both producer and server side are
> > >
> > > turned
> > >
> > > on.
> > >
> > > Its exactly the same behavior in KIP-361.
> > >
> > > I suggest running a streams job as a test. It creates the topics it
> > >
> > > needs
> > >
> > > using AdminClient. The value of auto.topic.create.enable is not
> > >
> > > relevant.
> > >
> > > Whether it is true or false, the topics will still be created.
> > >
> > > "In general, it would be nice if we could keep the security and
> > >
> > > access
> > >
> > > control model simple and not introduce a lot of special cases and
> > > exceptions. Kafka has basically converged on using ACLs and
> > > CreateTopicPolicy classes to control who can create topics and
> > >
> > > where.
> > >
> > > Adding more knobs seems like a step backwards, especially when the
> > >
> > > proposed
> > >
> > > knobs don't work consistently across components, and don't provide
> > >
> > > true
> > >
> > > security." This is not about access control at all. Shipping sane
> > >
> > > defaults
> > >
> > > should be prioritized.
> > >
> > > I don't think the default is really the question here. I think
> > >
> > > everyone
> > >
> > > agrees that the default for client-side auto-topic creation should
> > >
> > > be
> > >
> > > off.
> > >
> > > My point goes back to KIP-361 why we kept the server side config to
> > >
> > > have
> > >
> > > preference. We can keep bringing back the discussion of security
> > >
> > > policies
> > >
> > > but a producer client overiding server side setting is not just
> > >
> > > security
> > >
> > > policy issue .
> > >
> > > "Producer client can override server side setting and not consumer
> > >
> > > and
> > >
> > > also producer can only override creation of topic but no the
> > >
> > > replication
> > >
> > > and partitions. " This doesn't make sense to me and why we want to
> > > introduce such a confusing behavior.
> > >
> > > The scenarios that you're describing (such as dealing with a poorly
> > > configured client that tries to create topics it should not) do
> > >
> > > seem
> > >
> > > to
> > >
> > > be
> > >
> > > about access control.
> > >
> > > We keep talking about CreateTopicPolicy and yet we don't have
> > >
> > > default
> > >
> > > one
> > >
> > > and asking every user of Kafka implement their own doesn't make
> > >
> > > sense
> > >
> > > here.
> > >
> > > The point of CreateTopicPolicy is exactly that you can implement
> > >
> > > your
> > >
> > > own,
> > >
> > > though. It's a way of customizing what the policy is in your
> > >
> > > specific
> > >
> > > cluster.
> > >
> > > I agree that most people don't want to write a custom policy. But
> > >
> > > most
> > >
> > > of
> > >
> > > those people are satisfied with just setting up ACLs to set up
> > >
> > > simple
> > >
> > > policies like this user can create topics, that user can't, etc.
> > >
> > > That's
> > >
> > > why
> > >
> > > I suggested adding support for ACLs to insecure clusters might be
> > >
> > > helpful.
> > >
> > > Also, just as a side note, wouldn't it would be impossible for us
> > >
> > > to
> > >
> > > specify a default CreateTopicsPolicy that did anything at this
> > >
> > > point
> > >
> > > anyway
> > >
> > > without breaking backwards compatibility? Maybe I'm
> > >
> > > misunderstanding
> > >
> > > the
> > >
> > > proposal.
> > >
> > > I am asking about exact behavior that we shipped for consumer side
> > >
> > > https:/
> > >
> > > / cwiki. apache. org/ confluence/ display/ KAFKA/
> > >
> > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > >
> > >
> > > (
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > >
> > >
> > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > >
> > > )
> > >
> > > I still didn't get any response why this behavior shouldn't be
> > >
> > > exactly
> > >
> > > like Kafka consumer and why do we want to have producer to
> > >
> > > overrider
> > >
> > > server
> > >
> > > side config and while not allowing consumer to do so. We are not
> > >
> > > even
> > >
> > > allowing the same contract and this will cause more confusion from
> > >
> > > the
> > >
> > > users standpoint.
> > >
> > > Hmm. The consumer already has a way to override the server side
> > > configuration. See KIP-361: Add Consumer Configuration to Disable
> > >
> > > Auto
> > >
> > > Topic Creation. This lets the consumer skip auto-creating topics,
> > >
> > > even
> > >
> > > if
> > >
> > > auto-creation is enabled on the broker.
> > >
> > > Yes that's what I am saying and it doesn't allow topic creation if
> > >
> > > the
> > >
> > > server side auto-creation is disabled and in consumer its enabled.
> > >
> > > I
> > >
> > > would like to see the exact behavior for producer as stated in
> > >
> > > KIP-361.
> > >
> > > To be fair, the KIP should probably discuss why we don't want to
> > >
> > > implement
> > >
> > > client-side topic creation in the consumer in "rejected
> > >
> > > alternatives."
> > >
> > > Maybe Justine can add more context here in the KIP.
> > >
> > > The last time we talked about this, the reasoning was that we
> > >
> > > wanted
> > >
> > > to
> > >
> > > eventually get rid of consumer-side auto-topic creation entirely,
> > >
> > > and
> > >
> > > so
> > >
> > > it
> > >
> > > wasn't worth implementing an improved way of doing it. Producer
> > >
> > > side
> > >
> > > auto
> > >
> > > topic-creation, in contrast, will probably stick around, although
> > >
> > > we'd
> > >
> > > like
> > >
> > > to deprecate and remove the problematic server-side mechanism over
> > >
> > > time.
> > >
> > > We should implement the producer side topic creation with proper
> > >
> > > create
> > >
> > > topic request and I am in favor of this KIP.
> > >
> > > All I am asking is don't make the producer to override server side
> > >
> > > config
> > >
> > > and keep it similar to KIP-361 just like consumer, it honors what
> > >
> > > set
> > >
> > > on
> > >
> > > server side which takes the precedence. Changing this behavior in
> producer
> > > is backward incompatible and will catch users by surprise.
> > >
> > > All arguments for enforcing producer side overriding config can
> > >
> > > still
> > >
> > > be
> > >
> > > achieved. Server side auto creation turned on by default and any
> > >
> > > new
> > >
> > > producer client setting their auto creation config to true will
> > >
> > > create
> > >
> > > topics in default behavior and any user who set server side to
> > >
> > > false
> > >
> > > and
> > >
> > > upgrades to latest kafka with this KIP will not see any unwanted
> > >
> > > behavior
> > >
> > > from clients. IMO this is not a unreasonable request on this KIP
> > >
> > > and
> > >
> > > this
> > >
> > > requested behavior is exactly what the KIP initially proposed.
> > >
> > > Thanks,
> > >
> > > Harsha
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache.
> > >
> > > org (
> > >
> > > cmccabe@apache.org ) > wrote:
> > >
> > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > >
> > > Not sure how the AdminClient applies here, It is an external API
> > >
> > > and
> > >
> > > is
> > >
> > > not part of KafkaProducer so any user who updates to latest version
> > >
> > > of
> > >
> > >
> > > Kafka with this feature get to create the topics.
> > > They have to build a tooling around AdminClient allow themselves to
> > >
> > >
> > > create
> > >
> > > topics.
> > >
> > > Hi Harsha,
> > >
> > > Hmm... I'm not sure I follow. Users don't have to build their own
> > >
> > > tooling,
> > >
> > > right? They can use any of the shell scripts that we've shipped in
> > >
> > > the
> > >
> > > last
> > >
> > > few releases. For example, if any of your users run it, this shell
> > >
> > > script
> > >
> > > will delete all of the topics from your non-security-enabled
> > >
> > > cluster:
> > >
> > >
> > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --delete
> > > --topic
> > >
> > >
> > > They will need to fill in the correct bootstrap servers list, of
> > >
> > > course,
> > >
> > > not localhost. This deletion script will work on some pretty old
> > >
> > > brokers,
> > >
> > > even back to the 0.10 releases. It seems a little odd to trust your
> > >
> > > users
> > >
> > > with this power, but not trust them to avoid changing a particular
> > > configuration key.
> > >
> > > There is no behavior in Kafka producer that allowed users to delete
> > >
> > > the
> > >
> > > topics or delete the records. So citing them as an example doesn't
> > >
> > > makes
> > >
> > > sense in this context.
> > >
> > > I think Kafka Streams is relevant here. After all, it's software
> > >
> > > that
> > >
> > > we
> > >
> > > ship as part of the official Kafka release. And it auto-creates
> > >
> > > topics
> > >
> > > even
> > >
> > > when auto.create.topics.enable is set to false on the broker.
> > >
> > > I think that auto.create.topics.enable was never intended as a
> > >
> > > security
> > >
> > > setting to control access. It was intended as a way to disable the
> > > broker-side auto-create feature specifically, because there were
> > >
> > > some
> > >
> > > problems with that specific feature. Broker-side auto-creation is
> > > frustrating because it's on by default, and because it can
> > >
> > > auto-create
> > >
> > > topics even if the producer or consumer didn't explicitly ask for
> > >
> > > that.
> > >
> > > Neither of those problems applies to this KIP: producers have to
> > > specifically opt in, and it won't be on by default. Basically, we
> > >
> > > think
> > >
> > > that client-side auto-creation is actually a lot better-- hence
> > >
> > > this
> > >
> > > KIP
> > >
> > > :)
> > >
> > >
> > > But there is
> > > a functionality which allowed creation of topics if they don't
> > >
> > >
> > > exist
> > >
> > > in
> > >
> > > the
> > >
> > > cluster and this behavior could be controlled by a config on the
> > >
> > > server
> > >
> > > side. Now with this KIP we are allowing producer to make that
> > >
> > > decision
> > >
> > > without any gateway on the server via configs. Any user who is not
> > >
> > > aware
> > >
> > > of
> > >
> > >
> > > such changes
> > > can accidentally create these topics and we are essentially
> > >
> > >
> > > removing
> > >
> > > a
> > >
> > > config that exists in brokers today to block this accidental
> > >
> > > creation
> > >
> > > and
> > >
> > > allowing clients to take control.
> > >
> > > Again, I hope I'm not misinterpreting, but I don't see how this can
> > >
> > > be
> > >
> > > turned on accidentally. The user would have to specifically turn
> > >
> > > this
> > >
> > > on
> > >
> > > in
> > >
> > > the producer by setting the configuration key.
> > >
> > > I still didn't get any positives of not having server side configs?
> > >
> > > if
> > >
> > > you
> > >
> > > want to turn them on and allow any client to create topics set the
> > >
> > > default
> > >
> > >
> > > to true
> > > and allow users who doesn't want to have this feature let them turn
> > >
> > >
> > > them
> > >
> > > off. It will be the exact behavior as it is today , as far as
> > >
> > > producer
> > >
> > > is
> > >
> > >
> > > concerned. I am not
> > > understanding why not having server side configs to gateways such a
> > >
> > >
> > > hard
> > >
> > > requirement and this behavior exists today. As far I am concerned
> > >
> > > this
> > >
> > > breaks the backward compatibility.
> > >
> > > The downside is that if we wanted to check a server side
> > >
> > > configuration
> > >
> > > before sending the create topics request, the code would be more
> > >
> > > complex.
> > >
> > > The behavior would also not be consistent with how topic
> > >
> > > auto-creation
> > >
> > > is
> > >
> > > handled in Kafka Streams.
> > >
> > > In general, it would be nice if we could keep the security and
> > >
> > > access
> > >
> > > control model simple and not introduce a lot of special cases and
> > > exceptions. Kafka has basically converged on using ACLs and
> > > CreateTopicPolicy classes to control who can create topics and
> > >
> > > where.
> > >
> > > Adding more knobs seems like a step backwards, especially when the
> > >
> > > proposed
> > >
> > > knobs don't work consistently across components, and don't provide
> > >
> > > true
> > >
> > > security.
> > >
> > > Maybe we should support an insecure mode where there are still
> > >
> > > users
> > >
> > > and
> > >
> > > ACLs. That would help people who don't want to set up Kerberos,
> > >
> > > etc.
> > >
> > > but
> > >
> > > still want to protect against misconfigured clients or accidents.
> > >
> > > Hadoop
> > >
> > > has something like this, and I think it was useful. NFS also
> > >
> > > supported
> > >
> > > (supports?) a mode where you just pass whatever user ID you want
> > >
> > > and
> > >
> > > the
> > >
> > > system believes you. These things clearly don't protect against
> > >
> > > malicious
> > >
> > > users, but they can help set up policies when needed.
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache.
> > >
> > > org (
> > >
> > > cmccabe@apache.org ) > wrote:
> > >
> > > Hi Harsha,
> > >
> > > Thanks for taking a look.
> > >
> > > I'm not sure I follow the discussion about AdminClient.
> > >
> > > KafkaAdminClient
> > >
> > > has been around for about 2 years at this point as a public class.
> > >
> > > There
> > >
> > > are many programs that use it to automatically create topics. Kafka
> > > Streams does this, for example. If any of your users run Kafka
> > >
> > > Streams,
> > >
> > > they will be auto-creating topics, regardless of what setting you
> > >
> > > use
> > >
> > > for
> > >
> > > auto.create.topics.enable.
> > >
> > > A big part of the annoyance of auto-topic creation right now is
> > >
> > > that
> > >
> > > it is
> > >
> > > on by default. The new configuration proposed by KIP-487 wouldn't
> > >
> > > be.
> > >
> > > Users would have to explicitly opt in to the new behavior of
> > >
> > > client-side
> > >
> > > topic creation. If you run without security, you're already
> > >
> > > putting a
> > >
> > > huge
> > >
> > > amount of trust in your users. For example, you trust them not to
> > >
> > > delete
> > >
> > > records with the kafka-delete-records. sh (
> > >
> > > http://kafka-delete-records.
> > >
> > >
> > > sh/
> > > ) command, or delete topics with kafka-topics. sh (
> > >
> > >
> > > http://kafka-topics.
> > >
> > >
> > > sh/
> > > ). Trusting them not to set a certain config value seems minor in
> > > comparison, right?
> > >
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > >
> > >
> > > Hi,
> > > Even with policies one needs to implement that, so for every
> > >
> > >
> > > user who
> > >
> > > doesn't want a producer to create topics or have limits around
> > >
> > > partitions &
> > >
> > > replication factor they have to implement a policy. The KIP is
> > >
> > > changing
> > >
> > > the behavior , it might not be introducing
> > >
> > > the
> > >
> > > new functionality but it will enable producers to override the
> > >
> > > create
> > >
> > > topic
> > >
> > > config settings on the broker. What I am asking for to provide a
> > >
> > > config
> > >
> > > that will disable auto creation of topics and if its enabled set
> > >
> > > some
> > >
> > > sane
> > >
> > > defaults so that clients can create a topic with in those limits. I
> > >
> > > don't
> > >
> > >
> > > see how this not related to this KIP.
> > > If the server config options as I suggested doesn't interest
> > >
> > >
> > > you at
> > >
> > >
> > > least have a default CreateTopicPolices in place.
> > > To give an example, In our environment we disable the
> > > auto.create.topic.enable and force users to go through a
> > >
> > >
> > > centralized
> > >
> > > service as we want capture more details about the topic creation
> > >
> > > and
> > >
> > > requirements. With this KIP, a producer can create a topic with no
> > >
> > > bounds.
> > >
> > > Another example max.message.size we define that at cluster level
> > >
> > > and one
> > >
> > > can override max.messsage.size at topic level, any misconfiguration
> > >
> > > at
> > >
> > > this
> > >
> > > will cause service degradation. Its not always about the rogue
> > >
> > > clients,
> > >
> > > users can easily misconfigure and can cause an outage. Again we can
> > >
> > > talk
> > >
> > > about CreateTopicPolicy but without having a
> > >
> > > default
> > >
> > > implementation and asking everyone to implement their own while
> > >
> > > changing
> > >
> > > the behavior in producer doesn't make sense to me.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me.
> > >
> > > uk (
> > >
> > > ismael@juma.me.uk ) >
> > >
> > > wrote:
> > >
> > > Hi Harsha,
> > >
> > > I mentioned policies and the authorizer. For example, with
> > > CreateTopicPolicy, you can implement the limits you describe. If
> > >
> > > you
> > >
> > > have
> > >
> > > ideas of how that should be improved, please submit a KIP. My
> > >
> > > point is
> > >
> > > that
> > >
> > > this KIP is not introducing any new functionality with regards to
> > >
> > > what
> > >
> > > rogue clients can do. It's using the existing protocol that is
> > >
> > > already
> > >
> > > exposed via the AdminClient. So, I don't think we need to address
> > >
> > > it in
> > >
> > > this KIP. Does that make sense?
> > >
> > > Ismael
> > >
> > > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> > >
> > > kafka@ harsha. io ( kafka@harsha.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Ismael,
> > > Sure AdminClient can do that and we should've shipped a config or
> > >
> > >
> > > use
> > >
> > > the
> > >
> > > existing one to block that. Not all users are yet to upgrade to
> > >
> > > AdminClient
> > >
> > > and start using that to cause issues yet. In shared environment we
> > >
> > > should
> > >
> > > allow server to set sane defaults and not allow every client to go
> > >
> > > ahead
> > >
> > > create random no.of topic/partitions and replication factor. Even
> > >
> > > if
> > >
> > > the
> > >
> > > users want to allow topic creation proposed in the KIP , it makes
> > >
> > > sense to
> > >
> > > have some guards against the no.of partitions and replication
> > >
> > > factor.
> > >
> > > Authorizer is not always an answer to block requests and having
> > >
> > > users
> > >
> > > set
> > >
> > > server side configs to protect a multi-tenant environment is
> > >
> > > required.
> > >
> > > In a
> > >
> > > non-secure environment Authorizer is a blunt instrument either you
> > >
> > > end
> > >
> > > up
> > >
> > >
> > > blocking everyone or allowing everyone.
> > > I am asking to have server side that allow clients to create
> > >
> > >
> > > topics or
> > >
> > > not
> > >
> > > , if they are allowed set a ceiling on max no.of partitions and
> > > replication-factor.
> > >
> > > -Harsha
> > >
> > > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com (
> > >
> > > ismaelj@gmail.com
> > >
> > > )
> > >
> > > wrote:
> > >
> > > Harsha,
> > >
> > > Rogue clients can use the admin client to create topics and
> > >
> > > partitions.
> > >
> > > ACLs and policies can help in that case as well as this one.
> > >
> > > Ismael
> > >
> > > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha.
> > >
> > > io
> > >
> > > (
> > >
> > > kafka@harsha.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Hi Justine,
> > > Thanks for the KIP.
> > > "When server-side auto-creation is disabled, client-side
> > >
> > >
> > > auto-creation
> > >
> > >
> > > will try to use client-side configurations"
> > > If I understand correctly, this KIP is removing any server-side
> > >
> > >
> > > blocking
> > >
> > >
> > > client auto creation of topic?
> > > if so this will present potential issue of rogue client creating
> > >
> > >
> > > ton of
> > >
> > > topic-partitions and potentially bringing down the service for
> > >
> > > everyone
> > >
> > > or
> > >
> > >
> > > degrade the service itself.
> > > By reading the KIP its not clear to me that there is a clear way to
> > >
> > >
> > > block
> > >
> > > auto creation topics of all together from clients by server side
> > >
> > > config.
> > >
> > > Server side configs of default topic, partitions should take higher
> > > precedence and client shouldn't be able to create a topic with
> > >
> > > higher
> > >
> > > no.of
> > >
> > > partitions, replication than what server config specifies.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Hi all,
> > > I made some changes to the KIP. Hopefully this configuration change
> > >
> > >
> > > will
> > >
> > > make things a little clearer.
> > >
> > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > >
> > > https://cwiki.
> > >
> > >
> > > apache.org/confluence/display/KAFKA/ )
> > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > >
> > >
> > > Please let me know if you have any feedback or questions!
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache.
> > >
> > > org (
> > >
> > > cmccabe@apache.org ) >
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think you bring up a good point. It would be better if we didn't
> > >
> > > ever
> > >
> > > have to set up client-side configuration for this feature, and
> > >
> > > KIP-464
> > >
> > > would let us skip this entirely.
> > >
> > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > >
> > >
> > > Hi Mickael,
> > > I agree that KIP-464 works on newer brokers, but I was a bit
> > >
> > >
> > > worried
> > >
> > > how
> > >
> > > things would play out on older brokers that* do not *have KIP 464
> > >
> > > included.
> > >
> > > Is it enough to throw an error in this case when producer configs
> > >
> > > are
> > >
> > > not
> > >
> > > specified?
> > >
> > > I think the right thing to do would be to log an error message in
> > >
> > > the
> > >
> > > client. We will need to have that capability in any case, to cover
> > > scenarios like the client trying to auto-create a topic that they
> > >
> > > don't
> > >
> > > have permission to create. Or a client trying to create a topic on
> > >
> > > a
> > >
> > > broker
> > >
> > > so old that CreateTopicsRequest is not supported.
> > >
> > > The big downside to relying on KIP-464 is that it is a very recent
> > >
> > > feature
> > >
> > > -- so recent that it hasn't even made its way to any official
> > >
> > > Apache
> > >
> > > release. It's scheduled for the upcoming 2.4 release in a few
> > >
> > > months.
> > >
> > > So if you view this KIP as a step towards removing broker-side
> > > auto-create, you might want to support older brokers just to
> > >
> > > accelerate
> > >
> > > adoption, and hasten the day when we can finally flip broker-side
> > > auto-create to off (or even remove it entirely).
> > >
> > > I have to agree, though, that having client-side configurations for
> > >
> > > number
> > >
> > > of partitions and replication factor is messy. Maybe it would be
> > >
> > > worth
> > >
> > > it
> > >
> > > to restrict support to post-KIP-464 brokers, if we could avoid
> > >
> > > creating
> > >
> > > more configs.
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> > >
> > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > We can rely on KIP-464 which allows to omit the partition count or
> > > replication factor when creating a topic. In that case, the broker
> > >
> > > defaults
> > >
> > > are used.
> > >
> > > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Michael,
> > > That makes sense to me!
> > > To clarify, in the current state of the KIP, the producer does not
> > >
> > >
> > > rely
> > >
> > > on
> > >
> > > the broker to autocreate--if the broker's config is disabled, then
> > >
> > > the
> > >
> > > producer can autocreate on its own with a create topic request (the
> > >
> > > same
> > >
> > >
> > > type of request the admin client uses).
> > > However, if both configs are enabled, the broker will autocreate
> > >
> > >
> > > through
> > >
> > > a
> > >
> > > metadata request before the producer gets a chance. Of course, the
> > >
> > > way
> > >
> > > to
> > >
> > > avoid this, is to do as you suggested, and set
> > >
> > > the
> > >
> > > "allow_auto_topic_creation" field to false.
> > >
> > > I think the only thing we need to be careful with in this setup is
> > >
> > > without
> > >
> > > KIP 464, we can not use broker defaults for this topic. A user
> > >
> > > needs
> > >
> > > to
> > >
> > > specify the number of partition and replication factor in the
> > >
> > > config.
> > >
> > > An
> > >
> > > alternative to this is to have coded defaults for when these
> > >
> > > configs
> > >
> > > are
> > >
> > > unspecified, but it is not immediately apparent what these defaults
> > >
> > > should
> > >
> > > be.
> > >
> > >
> > > Thanks again for reading my KIP,
> > > Justine
> > >
> > >
> > > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> > >
> > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > >
> > > Thanks for the response!
> > > In my opinion, it would be better if the producer did not rely at
> > >
> > >
> > > all
> > >
> > >
> > > on the broker auto create feature as this is what we're aiming to
> > > deprecate. When requesting metadata we can set the
> > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > creation. Then if the topic is not existing, send a
> > >
> > >
> > > CreateTopicRequest.
> > >
> > > What do you think?
> > >
> > > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Currently the way it is implemented, the broker auto-creation
> > >
> > > configuration
> > >
> > > takes precedence. The producer will not use the CreateTopics
> > >
> > > request.
> > >
> > > (Technically it can--but the topic will already be created
> > >
> > > through
> > >
> > > the
> > >
> > > broker, so it will never try to create the topic.) It is possible
> > >
> > > to
> > >
> > > change this however, and I'd be happy to
> > >
> > > discuss
> > >
> > > the
> > >
> > > benefits of this alternative.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> > >
> > > mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP!
> > >
> > > In case auto creation is enabled on both the client and server,
> > >
> > > will
> > >
> > > the producer still use the AdminClient (CreateTopics request)
> > >
> > > to
> > >
> > > create topics? and not rely on the broker auto create. I'm
> > >
> > > guessing the
> > >
> > > answer is yes but can you make it explicit.
> > >
> > > Thank you
> > >
> > > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Hi,
> > > Just a friendly reminder to take a look at this KIP if you
> > >
> > >
> > > have
> > >
> > > the
> > >
> > > time.
> > >
> > > I was thinking about broker vs. client default precedence,
> > >
> > > and I
> > >
> > > think it
> > >
> > > makes sense to keep the broker as the default used when both
> > >
> > > client-side
> > >
> > > and broker-side defaults are configured. The idea is that
> > >
> > > there
> > >
> > > would be
> > >
> > > pretty clear documentation, and that many systems with
> > >
> > > configurations
> > >
> > > that
> > >
> > > the client could not change would likely have the auto-create
> > >
> > > default
> > >
> > > off.
> > >
> > > (In cloud for example).
> > >
> > >
> > > It also seems like in most cases, the consumer config
> > > ' allow. auto. create. topics ( http://allow.auto.create.topics/
> > >
> > >
> > > ) '
> > >
> > > was
> > >
> > > created to actually prevent
> > >
> > > the
> > >
> > > creation
> > >
> > > of
> > >
> > > topics, so the loss of creation functionality will not be a
> > >
> > > big
> > >
> > > problem.
> > >
> > > I'm happy to discuss any other compatibility problems or
> > >
> > > components
> > >
> > > of
> > >
> > > this KIP.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io )
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I was looking at this KIP again, and there is a decision I
> > >
> > > made
> > >
> > > that I
> > >
> > > think is worth discussing.
> > >
> > >
> > > In the case where both the broker and producer's
> > > 'auto.create.topics.enable' are set to true, we have to
> > >
> > >
> > > choose
> > >
> > > either
> > >
> > > the
> > >
> > > broker configs or the producer configs for the replication
> > > factor/partitions.
> > >
> > > Currently, the decision is to have the broker defaults take
> > >
> > > precedence.
> > >
> > > (It is easier to do this in the implementation.) It also
> > >
> > > makes
> > >
> > > some
> > >
> > > sense
> > >
> > > for this behavior to take precedence since this behavior
> > >
> > > already
> > >
> > > occurs as
> > >
> > > the default.
> > >
> > > However, I was wondering if it would be odd for those who
> > >
> > > can
> > >
> > > only
> > >
> > > see
> > >
> > > the
> > >
> > > producer side to set configs for replication
> > >
> > > factor/partitions
> > >
> > > and
> > >
> > > see
> > >
> > > different results. Currently the documentation for the
> > >
> > > config
> > >
> > > states
> > >
> > > that
> > >
> > > the config values are only used when the broker config is
> > >
> > > not
> > >
> > > enabled,
> > >
> > > but
> > >
> > > this might not always be clear to the user. Changing the
> > >
> > > code
> > >
> > > to
> > >
> > > have
> > >
> > > the
> > >
> > > producer's configurations take precedence is possible, but
> > >
> > > I
> > >
> > > was
> > >
> > > wondering
> > >
> > > what everyone thought.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Just a quick update--
> > >
> > > It seems that enabling both the broker and producer
> > >
> > > configs
> > >
> > > works
> > >
> > > fine,
> > >
> > > except that the broker configurations for partitions,
> > >
> > > replication
> > >
> > > factor
> > >
> > >
> > > take precedence.
> > > I don't know if that is something we would want to
> > >
> > >
> > > change, but
> > >
> > > I'll be
> > >
> > > updating the KIP for now to reflect this. Perhaps we would
> > >
> > > want to
> > >
> > > add more
> > >
> > > to the documentation of the the producer configs to
> > >
> > > clarify.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for looking at the KIP. I can definitely add to
> > >
> > > the
> > >
> > > title
> > >
> > > to
> > >
> > > make
> > >
> > > it more clear.
> > >
> > > It makes sense that both configurations could be turned
> > >
> > > on
> > >
> > > since
> > >
> > > there
> > >
> > > are many cases where the user can not control the
> > >
> > > server-side
> > >
> > > configurations. I was a little concerned about how both
> > >
> > > interacting
> > >
> > > would
> > >
> > > work out -- if there would be to many requests for new
> > >
> > > topics,
> > >
> > > for
> > >
> > > example.
> > >
> > > But it since it does make sense to allow both
> > >
> > > configurations
> > >
> > > enabled, I
> > >
> > > will test out some scenarios and I'll change the KIP to
> > >
> > > support
> > >
> > > this.
> > >
> > > I also agree with documentation about distinguishing the
> > >
> > > differences. I
> > >
> > > was having some trouble with the wording but I like the
> > >
> > > phrases
> > >
> > > "server-side" and "client-side." That's a good
> > >
> > > distinction I
> > >
> > > can
> > >
> > > use
> > >
> > > when
> > >
> > > describing.
> > >
> > > I'll try to update the KIP soon keeping everyone's input
> > >
> > > in
> > >
> > > mind.
> > >
> > >
> > > Thanks,
> > > Justine
> > >
> > >
> > > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> > >
> > > cmccabe@ apache. org ( cmccabe@apache.org )
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP. This seems like a good step towards
> > >
> > > removing
> > >
> > > server-side topic auto-creation.
> > >
> > > We should add included "client-side" to the title of
> > >
> > > the KIP
> > >
> > > somewhere,
> > >
> > > to make it clear that we're talking about client-side
> > >
> > > auto
> > >
> > > creation.
> > >
> > > The KIP says:
> > >
> > > In order to automatically create topics with the
> > >
> > > producer, the
> > >
> > > producer's
> > >
> > > auto.create.topics.enable config must be set to true
> > >
> > > and
> > >
> > > the
> > >
> > > broker
> > >
> > > config should be set to false
> > >
> > > From a user's point of view, this seems
> > >
> > > counter-intuitive.
> > >
> > > In
> > >
> > > order to
> > >
> > > auto-create topics the broker's
> > >
> > > auto.create.topics.enable
> > >
> > > config
> > >
> > > should be
> > >
> > > set to false? It seems like the server-side
> > >
> > > auto-create is
> > >
> > > unrelated to
> > >
> > > the client-side auto-create. We could have both turned
> > >
> > > on
> > >
> > > (and
> > >
> > > I'm
> > >
> > > sure
> > >
> > > that in the real world, people will try this
> > >
> > > configuration...)
> > >
> > > There's no
> > >
> > > reason not to support this, I think.
> > >
> > > We should add some documentation explaining the
> > >
> > > difference
> > >
> > > between
> > >
> > > server-side and client-side auto-creation. Without
> > >
> > > documentation,
> > >
> > > an admin
> > >
> > > might think that they had disabled all forms of
> > >
> > > auto-creation by
> > >
> > > setting
> > >
> > > the -side setting to false-- but this is not the case,
> > >
> > > of
> > >
> > > course.
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > >
> > > Hi Dhruvil,
> > >
> > >
> > > Thanks for reading the KIP!
> > > That was the general idea for deprecation. We would
> > >
> > >
> > > log a
> > >
> > > warning
> > >
> > > when the
> > >
> > >
> > > config is enabled on the broker.
> > > I also believe that there would be a change to
> > >
> > >
> > > documentation.
> > >
> > > If there is anything else that should be done, please
> > >
> > > let
> > >
> > > me
> > >
> > > know!
> > >
> > > Justine
> > >
> > > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> > >
> > > dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP, this is great!
> > >
> > > Could you add some more information about what
> > >
> > > deprecating
> > >
> > > the
> > >
> > > broker
> > >
> > > configuration means? Would we log a warning in the
> > >
> > > logs
> > >
> > > when
> > >
> > > auto
> > >
> > > topic
> > >
> > > creation is enabled on the broker, for example?
> > >
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > >
> > > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I'd like to start a discussion thread for KIP-487. This KIP plans
> > >
> > > to
> > >
> > > deprecate the current system of
> > >
> > > auto-creating
> > >
> > > topics
> > >
> > > through requests to the metadata and give the
> > >
> > > producer the
> > >
> > > ability to
> > >
> > > automatically create topics instead.
> > >
> > > More information can be found here:
> > >
> > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > >
> > > https://cwiki.
> > >
> > >
> > > apache.org/confluence/display/KAFKA/ )
> > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > >
> > >
> > >
> > > Thank you,
> > > Justine Olshan
> > >
> > >
> > >
> >
>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Ismael Juma <is...@juma.me.uk>.

Hi Boyang,

"One thing I'm not fully convinced of is why we need to deprecate the
server side auto topic creation logic,"

The point is to eventually remove auto topic creation logic from
MetadataRequest. It makes no sense for MetadataRequest to cause topics to
be created. It's an unintuitive hack.

Ismael

On Tue, Jun 23, 2020 at 10:12 PM Boyang Chen <re...@gmail.com>
wrote:

> Hey Justin and Jiamei,
>
> I read the KIP and skimmed over the discussion. One thing I'm not fully
> convinced of is why we need to deprecate the server side auto topic
> creation logic, which seems orthogonal towards whether a client wants to
> create the topic or not. Won't it be more natural to assume that only when
> both server and client agree on turning on the switch, will a topic get
> created?
>
> Some clarifications would also be appreciated:
>
> 1. Could we include a link to KIP-464 and explain its relation to KIP-487?
> It's very hard to read through the proposal when readers only have a
> reference number to some KIP that is not briefed.
>
> 2. The KIP suggests, " In the producer, auto-creation of a topic will occur
> through a specific request rather than through a side effect of requesting
> metadata." Could we be specific such as whether we are going to introduce a
> separate RPC, or just send another CreateTopicRequest?
>
> Boyang
>
> On Wed, Jun 17, 2020 at 8:51 AM jiamei xie <ji...@gmail.com> wrote:
>
> > Hi all
> > For
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > ,  It has not been updated for a long time. And I made some update, which
> > has been pushed to https://github.com/apache/kafka/pull/8831
> >
> > MetadataRequest has method Builder(List<String> topics, boolean
> > allowAutoTopicCreation) by which we can set whether to enable
> > allowAutoTopicCreation from producer.
> > By default, allowAutoTopicCreation on Producer is true. And only if when
> > the allowAutoTopicCreation of Broker and Producer are true, the topic can
> > be auto-created.
> >
> > Besides, the test cases are changed:
> > There are 4 cases for brokerAutoTopicCreationEnable and
> > producerAutoCreateTopicsPolicy, Check if the topic is created under these
> > four cases.
> >      If brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy
> > are true:  assertTrue(topicCreated)
> >      else : intercept[ExecutionException]
> >
> > Looking forward to your feedback and comments. Thanks.
> >
> > Best wishes
> > Jiamei Xie
> >
> > On 2019/08/12 15:50:22, Harsha Chintalapani <ka...@harsha.io> wrote:
> > > On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > A few points:
> > > >
> > > > 1. I think the way backwards compatibility is being used here is not
> > > > correct. Any functionality that is only enabled if set via a config
> is
> > > > backwards compatible. People may disagree with the functionality or
> the
> > > > config, but it's not a backwards compatibility issue.
> > > >
> > >
> > > We are talking about both broker and producer as a  single entity and
> run
> > > by the same team/users. Allowing newer producer to create topics on a
> > older
> > > broker when auto.create.topics.enable set to false, breaks  server side
> > > contract that this config offered from the beginning.  IMO, it clearly
> > > isn't backward compatible. User who set auto.create.topic.enable on
> > broker
> > > will not be the same who will turn it on producer side .
> > >
> > >
> > > > 2. It's an interesting question if auto topic creation via the
> producer
> > > > should be a server driven choice or not. I can see the desire to
> have a
> > > > server-driven default, but it seems like this is often application
> > > > specific. Because the functionality is trivially available via
> > AdminClient
> > > > (released 2 years ago), it's not quite possible to control what
> > > > applications do without the use of ACLs or policies today.
> > > >
> > > >
> > > >
> > > Producers & consumers are the majority of the clients in Kafka
> ecosystem.
> > > Just because AdminClient shipped a while back that doesn't mean all
> users
> > > adopting to it. To this day lot more users are aware of Producer &
> > Consumer
> > > APIs and running them in production compare to AdminClient.
> > >
> > >
> > > > 3. Changing the create topics request in this way is highly
> > unintuitive in
> > > > my opinion and it relies on every client to pass the new field. For
> > > > example, if librdkafka added auto create functionality by relying on
> > their
> > > > AdminClient, it would behave differently than what is proposed here.
> > > > Forcing every client to implement this change when calling auto
> create
> > from
> > > > the producer specifically seems odd
> > > >
> > >
> > > I am not sure why its unintuitive , protocols change. We add or upgrade
> > the
> > > existing protocols all the time.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > > .
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Aug 8, 2019 at 11:51 AM Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > Hi, Justine,
> > > >
> > > > Thanks for the KIP. Overall, it seems to be a good improvement.
> > > >
> > > > However, I think Harsha's point seems reasonable. We had
> > > > auto.create.topics.enable config on the broker to allow admins to
> > disable
> > > > topic creation from the producer/consumer clients before we had the
> > > > security feature. The need for that config is reduced with the
> security
> > > > feature, but may still be present since not all places have security
> > > > enabled. It's true that a non-secured environment is vulnerable to
> some
> > > > additional attacks, but producer/consumer are the most common way
> for a
> > > > user to interact with the broker. So, keeping that config for
> backward
> > > > compatibility could still be useful if it's not introducing too much
> > effort
> > > > or extra confusion.
> > > >
> > > >
> > > > Here is a one potential alternative way that I was thinking. We add a
> > new
> > > > field in the CreateTopicRequest to indicate whether it's from the
> > producer
> > > > or not. If auto.create.topics.enable is false, CreateTopicRequest
> from
> > the
> > > > producer will be rejected. We probably don't need to introduce the
> new
> > > > config (which seems a bit hard to explain) in the producer. Instead,
> > the
> > > > new producer always uses MetadataRequest with AllowAutoTopicCreation
> > set to
> > > > false to get the metadata and if the metadata is not present, send
> the
> > new
> > > > CreateTopicRequest
> > > > (assuming the broker supports it) to try to create the topic
> > > > automatically. Whether the creation is allowed or not will be
> > determined by
> > > > the broker. This will make the behavior backward compatible and we
> can
> > > > still achieve the main goal of the KIP, which is not relying on
> > > > MetadataRequest for topic creation. What do you think?
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Aug 8, 2019 at 1:34 AM M. Manna <ma...@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > >
> > > > If I may, perhaps you could simplify everything by using only
> > > > 'auto.create.topics.enable' as a value along with true. In other
> words,
> > > >
> > > >
> > > > the
> > > >
> > > > public interfaces section should only have
> > > >
> > > > [true,auto.create.topics.enable,
> > > >
> > > > false].
> > > >
> > > > The reason for this is that auto.create.topics.enable is already
> known
> > to
> > > > users as a "Server-SIde" config. So all you are saying is
> > > >
> > > > a) To avoid day 1 impact, it will follow whatever
> > > >
> > > > auto.create.topics.enable
> > > >
> > > >
> > > > value is set.
> > > > b) False means - no client side topic creation
> > > > c) True means client side topic creation.
> > > >
> > > >
> > > > It saves creating 2 more new strings :). But not too expensive
> anyway.
> > > >
> > > > Also, when you deprecate auto.create.topics.enable - you must provide
> > > > sufficient logic to ensure that things like rolling upgrade doesn't
> > > > temporarily break anything. I apologise if you have already accounted
> > for
> > > > this, but wanted to mention since I didn't notice this on the KIP.
> > > >
> > > > Let me know how this sounds.
> > > >
> > > > Regards,
> > > >
> > > > On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jo...@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Hi Harsha,
> > > >
> > > > I think my message may have gotten lost in all the others.
> > > >
> > > > Two of the goals of this KIP are to 1) allow auto-creation on
> specific
> > > > clients when the broker default is false and 2) eventually replace
> the
> > > > broker config.
> > > >
> > > > In order to accomplish these two goals, we need the producer to be
> able
> > > >
> > > > to
> > > >
> > > >
> > > > create topics despite the broker config. (How can we replace a
> function
> > > > when we rely on it?)
> > > > I think at this point we have a fundamental disagreement in what we
> > > >
> > > >
> > > > should
> > > >
> > > >
> > > > allow the producer to do.
> > > > In my previous message I mentioned a config that would allow for the
> > > >
> > > >
> > > > broker
> > > >
> > > > to prevent producer auto-creation. (It would be disabled by default.)
> > > >
> > > > It
> > > >
> > > > would fix your issue for now, but could lead to more complications
> > > >
> > > > later.
> > > >
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > >
> > > > On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <kafka@harsha.io
> >
> > > > wrote:
> > > >
> > > > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz>
> > > >
> > > > wrote:
> > > >
> > > > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> > > >
> > > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe <
> > > >
> > > > cmccabe@apache.org
> > > >
> > > > wrote:
> > > >
> > > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> > > >
> > > >
> > > > Hi Colin,
> > > > "Hmm... I'm not sure I follow. Users don't have to build their own
> > > > tooling, right? They can use any of the shell scripts that we've
> > > >
> > > >
> > > > shipped
> > > >
> > > > in
> > > >
> > > > the last few releases. For example, if any of your users run it,
> > > >
> > > > this
> > > >
> > > > shell
> > > >
> > > > script will delete all of the topics from your non-security-enabled
> > > > cluster:
> > > >
> > > >
> > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > --bootstrap-server localhost:9092 --delete
> > > > --topic
> > > >
> > > >
> > > > They will need to fill in the correct bootstrap servers list, of
> > > >
> > > > course,
> > > >
> > > > not localhost. This deletion script will work on some pretty old
> > > >
> > > > brokers,
> > > >
> > > > even back to the 0.10 releases. It seems a little odd to trust your
> > > >
> > > > users
> > > >
> > > > with this power, but not trust them to avoid changing a particular
> > > > configuration key."
> > > >
> > > > The above will blocked by the server if we set delete.topic.enable
> > > >
> > > > to
> > > >
> > > > false and thats exactly what I am asking for.
> > > >
> > > > Hi Harsha,
> > > >
> > > > I was wondering if someone was going to bring up that configuration
> > > >
> > > > :)
> > > >
> > > > it's an interesting complication, but globally disabling topic
> > > >
> > > > deletion
> > > >
> > > > is
> > > >
> > > > not very practical for most use-cases.
> > > >
> > > > In any case, there are plenty of other bad things that users with
> > > >
> > > > full
> > > >
> > > > permissions can do that aren't blocked by any server configuration.
> > > >
> > > > For
> > > >
> > > > example, they can delete every record in every topic. I can write a
> > > >
> > > > script
> > > >
> > > > for that too, and there's no server configuration you can set to
> > > >
> > > > disable
> > > >
> > > > it. Or I could simply create hundreds of thousands of topics, until
> > > >
> > > > cluster
> > > >
> > > > performance becomes unacceptable (this will be even more of a
> > > >
> > > > problem
> > > >
> > > > if
> > > >
> > > > someone configured delete.topic.enable as false). Or publish bad
> > > >
> > > > data
> > > >
> > > > to
> > > >
> > > > every topic, etc. etc.
> > > >
> > > > The point I'm trying to make here is that you can't rely on these
> > > >
> > > > kind
> > > >
> > > > of
> > > >
> > > > server-side configurations for security. At most, they're a way to
> > > >
> > > > set
> > > >
> > > > up
> > > >
> > > > certain very simple policies. But the policies are so simple that
> > > >
> > > > they're
> > > >
> > > > hardly ever useful any more.
> > > >
> > > > For example, if the problem you want to solve is that you want a
> > > >
> > > > user
> > > >
> > > > to
> > > >
> > > > only be able to create 50 topics and not delete anyone else's
> > > >
> > > > topics,
> > > >
> > > > you
> > > >
> > > > can solve that with a CreateTopicsPolicy that limits the number of
> > > >
> > > > topics,
> > > >
> > > > and some ACLs. There's no combination of auto.create.topics.enable
> > > >
> > > > and
> > > >
> > > > delete.topic.enable that will help here.
> > > >
> > > > Hi Colin,
> > > >
> > > > Well you gave the example that a user can delete the
> > > >
> > > > topics
> > > >
> > > > just by running that script :).
> > > >
> > > > I understand there are open APIs in Kafka and can lead to rogue
> > > >
> > > > clients
> > > >
> > > > taking advantage of it without proper security in place.
> > > >
> > > > What I am asking so far in this thread is , this KIP is changing
> > > >
> > > > the
> > > >
> > > > producer behavior and its not backward compatible.
> > > >
> > > > The change is backwards compatible. The default will still be
> > > >
> > > > server-side
> > > >
> > > > topic auto-creation, just like now.
> > > >
> > > > You will have to specifically change the producer config to get the
> > > >
> > > > new
> > > >
> > > > behavior.
> > > >
> > > > I disagree. Today server can turn off the topic creation neither
> > > >
> > > > producer
> > > >
> > > > or consumer can create a topic. With this KIP , producer can create a
> > > >
> > > > topic
> > > >
> > > > by turning on client side config when server side config is turned
> > > >
> > > > off.
> > > >
> > > > We can still achieve
> > > >
> > > > the main goal of the KIP which is to change MetadataRequest
> > > >
> > > > creating
> > > >
> > > > topics and send a CreateTopicRequest from Producer and also keep
> > > >
> > > > the
> > > >
> > > > server
> > > >
> > > > side config to have precedence. This KIP originally written to
> > > >
> > > > have
> > > >
> > > > server
> > > >
> > > > side preference and there is not much explanation why it changed to
> > > >
> > > > have
> > > >
> > > > Producer side preference.
> > > >
> > > > Arguing that AdminClient can do that and so we are going to make
> > > >
> > > > Producer
> > > >
> > > > do the same doesn't make sense.
> > > >
> > > > "The downside is that if we wanted to check a server side
> > > >
> > > > configuration
> > > >
> > > > before sending the create topics request, the code would be more
> > > >
> > > > complex.
> > > >
> > > > The behavior would also not be consistent with how topic
> > > >
> > > > auto-creation
> > > >
> > > > is
> > > >
> > > > handled in Kafka Streams."
> > > >
> > > > I am not sure why you need to check server side configuration
> > > >
> > > > before
> > > >
> > > > sending create topics request. A user enables producer side config
> > > >
> > > > to
> > > >
> > > >
> > > > create topics.
> > > > Producer sends a request to the broker and if the broker has
> > > > auto.topic.create.enable to true (default) it will allow creation
> > > >
> > > >
> > > > of
> > > >
> > > > topics. If it set to false it returns error back to the client.
> > > >
> > > > auto.topic.create.enable has never affected CreateTopicsRequest. If
> > > >
> > > > you
> > > >
> > > > submit a CreateTopicsRequest and you are authorized, a topic will
> > > >
> > > > be
> > > >
> > > > created, regardless of what the value of auto.topic.create.enable
> > > >
> > > > is.
> > > >
> > > > This
> > > >
> > > > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I
> > > >
> > > > think?)
> > > >
> > > > I don't see how this behavior will be different in Kafka streams.
> > > >
> > > > By
> > > >
> > > > default server allows the topic creation and with this KIP, It will
> > > >
> > > > only
> > > >
> > > > allow creation of topic when both producer and server side are
> > > >
> > > > turned
> > > >
> > > > on.
> > > >
> > > > Its exactly the same behavior in KIP-361.
> > > >
> > > > I suggest running a streams job as a test. It creates the topics it
> > > >
> > > > needs
> > > >
> > > > using AdminClient. The value of auto.topic.create.enable is not
> > > >
> > > > relevant.
> > > >
> > > > Whether it is true or false, the topics will still be created.
> > > >
> > > > "In general, it would be nice if we could keep the security and
> > > >
> > > > access
> > > >
> > > > control model simple and not introduce a lot of special cases and
> > > > exceptions. Kafka has basically converged on using ACLs and
> > > > CreateTopicPolicy classes to control who can create topics and
> > > >
> > > > where.
> > > >
> > > > Adding more knobs seems like a step backwards, especially when the
> > > >
> > > > proposed
> > > >
> > > > knobs don't work consistently across components, and don't provide
> > > >
> > > > true
> > > >
> > > > security." This is not about access control at all. Shipping sane
> > > >
> > > > defaults
> > > >
> > > > should be prioritized.
> > > >
> > > > I don't think the default is really the question here. I think
> > > >
> > > > everyone
> > > >
> > > > agrees that the default for client-side auto-topic creation should
> > > >
> > > > be
> > > >
> > > > off.
> > > >
> > > > My point goes back to KIP-361 why we kept the server side config to
> > > >
> > > > have
> > > >
> > > > preference. We can keep bringing back the discussion of security
> > > >
> > > > policies
> > > >
> > > > but a producer client overiding server side setting is not just
> > > >
> > > > security
> > > >
> > > > policy issue .
> > > >
> > > > "Producer client can override server side setting and not consumer
> > > >
> > > > and
> > > >
> > > > also producer can only override creation of topic but no the
> > > >
> > > > replication
> > > >
> > > > and partitions. " This doesn't make sense to me and why we want to
> > > > introduce such a confusing behavior.
> > > >
> > > > The scenarios that you're describing (such as dealing with a poorly
> > > > configured client that tries to create topics it should not) do
> > > >
> > > > seem
> > > >
> > > > to
> > > >
> > > > be
> > > >
> > > > about access control.
> > > >
> > > > We keep talking about CreateTopicPolicy and yet we don't have
> > > >
> > > > default
> > > >
> > > > one
> > > >
> > > > and asking every user of Kafka implement their own doesn't make
> > > >
> > > > sense
> > > >
> > > > here.
> > > >
> > > > The point of CreateTopicPolicy is exactly that you can implement
> > > >
> > > > your
> > > >
> > > > own,
> > > >
> > > > though. It's a way of customizing what the policy is in your
> > > >
> > > > specific
> > > >
> > > > cluster.
> > > >
> > > > I agree that most people don't want to write a custom policy. But
> > > >
> > > > most
> > > >
> > > > of
> > > >
> > > > those people are satisfied with just setting up ACLs to set up
> > > >
> > > > simple
> > > >
> > > > policies like this user can create topics, that user can't, etc.
> > > >
> > > > That's
> > > >
> > > > why
> > > >
> > > > I suggested adding support for ACLs to insecure clusters might be
> > > >
> > > > helpful.
> > > >
> > > > Also, just as a side note, wouldn't it would be impossible for us
> > > >
> > > > to
> > > >
> > > > specify a default CreateTopicsPolicy that did anything at this
> > > >
> > > > point
> > > >
> > > > anyway
> > > >
> > > > without breaking backwards compatibility? Maybe I'm
> > > >
> > > > misunderstanding
> > > >
> > > > the
> > > >
> > > > proposal.
> > > >
> > > > I am asking about exact behavior that we shipped for consumer side
> > > >
> > > > https:/
> > > >
> > > > / cwiki. apache. org/ confluence/ display/ KAFKA/
> > > >
> > > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > >
> > > >
> > > > (
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > >
> > > >
> > > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > >
> > > > )
> > > >
> > > > I still didn't get any response why this behavior shouldn't be
> > > >
> > > > exactly
> > > >
> > > > like Kafka consumer and why do we want to have producer to
> > > >
> > > > overrider
> > > >
> > > > server
> > > >
> > > > side config and while not allowing consumer to do so. We are not
> > > >
> > > > even
> > > >
> > > > allowing the same contract and this will cause more confusion from
> > > >
> > > > the
> > > >
> > > > users standpoint.
> > > >
> > > > Hmm. The consumer already has a way to override the server side
> > > > configuration. See KIP-361: Add Consumer Configuration to Disable
> > > >
> > > > Auto
> > > >
> > > > Topic Creation. This lets the consumer skip auto-creating topics,
> > > >
> > > > even
> > > >
> > > > if
> > > >
> > > > auto-creation is enabled on the broker.
> > > >
> > > > Yes that's what I am saying and it doesn't allow topic creation if
> > > >
> > > > the
> > > >
> > > > server side auto-creation is disabled and in consumer its enabled.
> > > >
> > > > I
> > > >
> > > > would like to see the exact behavior for producer as stated in
> > > >
> > > > KIP-361.
> > > >
> > > > To be fair, the KIP should probably discuss why we don't want to
> > > >
> > > > implement
> > > >
> > > > client-side topic creation in the consumer in "rejected
> > > >
> > > > alternatives."
> > > >
> > > > Maybe Justine can add more context here in the KIP.
> > > >
> > > > The last time we talked about this, the reasoning was that we
> > > >
> > > > wanted
> > > >
> > > > to
> > > >
> > > > eventually get rid of consumer-side auto-topic creation entirely,
> > > >
> > > > and
> > > >
> > > > so
> > > >
> > > > it
> > > >
> > > > wasn't worth implementing an improved way of doing it. Producer
> > > >
> > > > side
> > > >
> > > > auto
> > > >
> > > > topic-creation, in contrast, will probably stick around, although
> > > >
> > > > we'd
> > > >
> > > > like
> > > >
> > > > to deprecate and remove the problematic server-side mechanism over
> > > >
> > > > time.
> > > >
> > > > We should implement the producer side topic creation with proper
> > > >
> > > > create
> > > >
> > > > topic request and I am in favor of this KIP.
> > > >
> > > > All I am asking is don't make the producer to override server side
> > > >
> > > > config
> > > >
> > > > and keep it similar to KIP-361 just like consumer, it honors what
> > > >
> > > > set
> > > >
> > > > on
> > > >
> > > > server side which takes the precedence. Changing this behavior in
> > producer
> > > > is backward incompatible and will catch users by surprise.
> > > >
> > > > All arguments for enforcing producer side overriding config can
> > > >
> > > > still
> > > >
> > > > be
> > > >
> > > > achieved. Server side auto creation turned on by default and any
> > > >
> > > > new
> > > >
> > > > producer client setting their auto creation config to true will
> > > >
> > > > create
> > > >
> > > > topics in default behavior and any user who set server side to
> > > >
> > > > false
> > > >
> > > > and
> > > >
> > > > upgrades to latest kafka with this KIP will not see any unwanted
> > > >
> > > > behavior
> > > >
> > > > from clients. IMO this is not a unreasonable request on this KIP
> > > >
> > > > and
> > > >
> > > > this
> > > >
> > > > requested behavior is exactly what the KIP initially proposed.
> > > >
> > > > Thanks,
> > > >
> > > > Harsha
> > > >
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache.
> > > >
> > > > org (
> > > >
> > > > cmccabe@apache.org ) > wrote:
> > > >
> > > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > > >
> > > > Not sure how the AdminClient applies here, It is an external API
> > > >
> > > > and
> > > >
> > > > is
> > > >
> > > > not part of KafkaProducer so any user who updates to latest version
> > > >
> > > > of
> > > >
> > > >
> > > > Kafka with this feature get to create the topics.
> > > > They have to build a tooling around AdminClient allow themselves to
> > > >
> > > >
> > > > create
> > > >
> > > > topics.
> > > >
> > > > Hi Harsha,
> > > >
> > > > Hmm... I'm not sure I follow. Users don't have to build their own
> > > >
> > > > tooling,
> > > >
> > > > right? They can use any of the shell scripts that we've shipped in
> > > >
> > > > the
> > > >
> > > > last
> > > >
> > > > few releases. For example, if any of your users run it, this shell
> > > >
> > > > script
> > > >
> > > > will delete all of the topics from your non-security-enabled
> > > >
> > > > cluster:
> > > >
> > > >
> > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > --bootstrap-server localhost:9092 --delete
> > > > --topic
> > > >
> > > >
> > > > They will need to fill in the correct bootstrap servers list, of
> > > >
> > > > course,
> > > >
> > > > not localhost. This deletion script will work on some pretty old
> > > >
> > > > brokers,
> > > >
> > > > even back to the 0.10 releases. It seems a little odd to trust your
> > > >
> > > > users
> > > >
> > > > with this power, but not trust them to avoid changing a particular
> > > > configuration key.
> > > >
> > > > There is no behavior in Kafka producer that allowed users to delete
> > > >
> > > > the
> > > >
> > > > topics or delete the records. So citing them as an example doesn't
> > > >
> > > > makes
> > > >
> > > > sense in this context.
> > > >
> > > > I think Kafka Streams is relevant here. After all, it's software
> > > >
> > > > that
> > > >
> > > > we
> > > >
> > > > ship as part of the official Kafka release. And it auto-creates
> > > >
> > > > topics
> > > >
> > > > even
> > > >
> > > > when auto.create.topics.enable is set to false on the broker.
> > > >
> > > > I think that auto.create.topics.enable was never intended as a
> > > >
> > > > security
> > > >
> > > > setting to control access. It was intended as a way to disable the
> > > > broker-side auto-create feature specifically, because there were
> > > >
> > > > some
> > > >
> > > > problems with that specific feature. Broker-side auto-creation is
> > > > frustrating because it's on by default, and because it can
> > > >
> > > > auto-create
> > > >
> > > > topics even if the producer or consumer didn't explicitly ask for
> > > >
> > > > that.
> > > >
> > > > Neither of those problems applies to this KIP: producers have to
> > > > specifically opt in, and it won't be on by default. Basically, we
> > > >
> > > > think
> > > >
> > > > that client-side auto-creation is actually a lot better-- hence
> > > >
> > > > this
> > > >
> > > > KIP
> > > >
> > > > :)
> > > >
> > > >
> > > > But there is
> > > > a functionality which allowed creation of topics if they don't
> > > >
> > > >
> > > > exist
> > > >
> > > > in
> > > >
> > > > the
> > > >
> > > > cluster and this behavior could be controlled by a config on the
> > > >
> > > > server
> > > >
> > > > side. Now with this KIP we are allowing producer to make that
> > > >
> > > > decision
> > > >
> > > > without any gateway on the server via configs. Any user who is not
> > > >
> > > > aware
> > > >
> > > > of
> > > >
> > > >
> > > > such changes
> > > > can accidentally create these topics and we are essentially
> > > >
> > > >
> > > > removing
> > > >
> > > > a
> > > >
> > > > config that exists in brokers today to block this accidental
> > > >
> > > > creation
> > > >
> > > > and
> > > >
> > > > allowing clients to take control.
> > > >
> > > > Again, I hope I'm not misinterpreting, but I don't see how this can
> > > >
> > > > be
> > > >
> > > > turned on accidentally. The user would have to specifically turn
> > > >
> > > > this
> > > >
> > > > on
> > > >
> > > > in
> > > >
> > > > the producer by setting the configuration key.
> > > >
> > > > I still didn't get any positives of not having server side configs?
> > > >
> > > > if
> > > >
> > > > you
> > > >
> > > > want to turn them on and allow any client to create topics set the
> > > >
> > > > default
> > > >
> > > >
> > > > to true
> > > > and allow users who doesn't want to have this feature let them turn
> > > >
> > > >
> > > > them
> > > >
> > > > off. It will be the exact behavior as it is today , as far as
> > > >
> > > > producer
> > > >
> > > > is
> > > >
> > > >
> > > > concerned. I am not
> > > > understanding why not having server side configs to gateways such a
> > > >
> > > >
> > > > hard
> > > >
> > > > requirement and this behavior exists today. As far I am concerned
> > > >
> > > > this
> > > >
> > > > breaks the backward compatibility.
> > > >
> > > > The downside is that if we wanted to check a server side
> > > >
> > > > configuration
> > > >
> > > > before sending the create topics request, the code would be more
> > > >
> > > > complex.
> > > >
> > > > The behavior would also not be consistent with how topic
> > > >
> > > > auto-creation
> > > >
> > > > is
> > > >
> > > > handled in Kafka Streams.
> > > >
> > > > In general, it would be nice if we could keep the security and
> > > >
> > > > access
> > > >
> > > > control model simple and not introduce a lot of special cases and
> > > > exceptions. Kafka has basically converged on using ACLs and
> > > > CreateTopicPolicy classes to control who can create topics and
> > > >
> > > > where.
> > > >
> > > > Adding more knobs seems like a step backwards, especially when the
> > > >
> > > > proposed
> > > >
> > > > knobs don't work consistently across components, and don't provide
> > > >
> > > > true
> > > >
> > > > security.
> > > >
> > > > Maybe we should support an insecure mode where there are still
> > > >
> > > > users
> > > >
> > > > and
> > > >
> > > > ACLs. That would help people who don't want to set up Kerberos,
> > > >
> > > > etc.
> > > >
> > > > but
> > > >
> > > > still want to protect against misconfigured clients or accidents.
> > > >
> > > > Hadoop
> > > >
> > > > has something like this, and I think it was useful. NFS also
> > > >
> > > > supported
> > > >
> > > > (supports?) a mode where you just pass whatever user ID you want
> > > >
> > > > and
> > > >
> > > > the
> > > >
> > > > system believes you. These things clearly don't protect against
> > > >
> > > > malicious
> > > >
> > > > users, but they can help set up policies when needed.
> > > >
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache.
> > > >
> > > > org (
> > > >
> > > > cmccabe@apache.org ) > wrote:
> > > >
> > > > Hi Harsha,
> > > >
> > > > Thanks for taking a look.
> > > >
> > > > I'm not sure I follow the discussion about AdminClient.
> > > >
> > > > KafkaAdminClient
> > > >
> > > > has been around for about 2 years at this point as a public class.
> > > >
> > > > There
> > > >
> > > > are many programs that use it to automatically create topics. Kafka
> > > > Streams does this, for example. If any of your users run Kafka
> > > >
> > > > Streams,
> > > >
> > > > they will be auto-creating topics, regardless of what setting you
> > > >
> > > > use
> > > >
> > > > for
> > > >
> > > > auto.create.topics.enable.
> > > >
> > > > A big part of the annoyance of auto-topic creation right now is
> > > >
> > > > that
> > > >
> > > > it is
> > > >
> > > > on by default. The new configuration proposed by KIP-487 wouldn't
> > > >
> > > > be.
> > > >
> > > > Users would have to explicitly opt in to the new behavior of
> > > >
> > > > client-side
> > > >
> > > > topic creation. If you run without security, you're already
> > > >
> > > > putting a
> > > >
> > > > huge
> > > >
> > > > amount of trust in your users. For example, you trust them not to
> > > >
> > > > delete
> > > >
> > > > records with the kafka-delete-records. sh (
> > > >
> > > > http://kafka-delete-records.
> > > >
> > > >
> > > > sh/
> > > > ) command, or delete topics with kafka-topics. sh (
> > > >
> > > >
> > > > http://kafka-topics.
> > > >
> > > >
> > > > sh/
> > > > ). Trusting them not to set a certain config value seems minor in
> > > > comparison, right?
> > > >
> > > >
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > > >
> > > >
> > > > Hi,
> > > > Even with policies one needs to implement that, so for every
> > > >
> > > >
> > > > user who
> > > >
> > > > doesn't want a producer to create topics or have limits around
> > > >
> > > > partitions &
> > > >
> > > > replication factor they have to implement a policy. The KIP is
> > > >
> > > > changing
> > > >
> > > > the behavior , it might not be introducing
> > > >
> > > > the
> > > >
> > > > new functionality but it will enable producers to override the
> > > >
> > > > create
> > > >
> > > > topic
> > > >
> > > > config settings on the broker. What I am asking for to provide a
> > > >
> > > > config
> > > >
> > > > that will disable auto creation of topics and if its enabled set
> > > >
> > > > some
> > > >
> > > > sane
> > > >
> > > > defaults so that clients can create a topic with in those limits. I
> > > >
> > > > don't
> > > >
> > > >
> > > > see how this not related to this KIP.
> > > > If the server config options as I suggested doesn't interest
> > > >
> > > >
> > > > you at
> > > >
> > > >
> > > > least have a default CreateTopicPolices in place.
> > > > To give an example, In our environment we disable the
> > > > auto.create.topic.enable and force users to go through a
> > > >
> > > >
> > > > centralized
> > > >
> > > > service as we want capture more details about the topic creation
> > > >
> > > > and
> > > >
> > > > requirements. With this KIP, a producer can create a topic with no
> > > >
> > > > bounds.
> > > >
> > > > Another example max.message.size we define that at cluster level
> > > >
> > > > and one
> > > >
> > > > can override max.messsage.size at topic level, any misconfiguration
> > > >
> > > > at
> > > >
> > > > this
> > > >
> > > > will cause service degradation. Its not always about the rogue
> > > >
> > > > clients,
> > > >
> > > > users can easily misconfigure and can cause an outage. Again we can
> > > >
> > > > talk
> > > >
> > > > about CreateTopicPolicy but without having a
> > > >
> > > > default
> > > >
> > > > implementation and asking everyone to implement their own while
> > > >
> > > > changing
> > > >
> > > > the behavior in producer doesn't make sense to me.
> > > >
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me.
> > > >
> > > > uk (
> > > >
> > > > ismael@juma.me.uk ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Harsha,
> > > >
> > > > I mentioned policies and the authorizer. For example, with
> > > > CreateTopicPolicy, you can implement the limits you describe. If
> > > >
> > > > you
> > > >
> > > > have
> > > >
> > > > ideas of how that should be improved, please submit a KIP. My
> > > >
> > > > point is
> > > >
> > > > that
> > > >
> > > > this KIP is not introducing any new functionality with regards to
> > > >
> > > > what
> > > >
> > > > rogue clients can do. It's using the existing protocol that is
> > > >
> > > > already
> > > >
> > > > exposed via the AdminClient. So, I don't think we need to address
> > > >
> > > > it in
> > > >
> > > > this KIP. Does that make sense?
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> > > >
> > > > kafka@ harsha. io ( kafka@harsha.io ) >
> > > >
> > > > wrote:
> > > >
> > > >
> > > > Ismael,
> > > > Sure AdminClient can do that and we should've shipped a config or
> > > >
> > > >
> > > > use
> > > >
> > > > the
> > > >
> > > > existing one to block that. Not all users are yet to upgrade to
> > > >
> > > > AdminClient
> > > >
> > > > and start using that to cause issues yet. In shared environment we
> > > >
> > > > should
> > > >
> > > > allow server to set sane defaults and not allow every client to go
> > > >
> > > > ahead
> > > >
> > > > create random no.of topic/partitions and replication factor. Even
> > > >
> > > > if
> > > >
> > > > the
> > > >
> > > > users want to allow topic creation proposed in the KIP , it makes
> > > >
> > > > sense to
> > > >
> > > > have some guards against the no.of partitions and replication
> > > >
> > > > factor.
> > > >
> > > > Authorizer is not always an answer to block requests and having
> > > >
> > > > users
> > > >
> > > > set
> > > >
> > > > server side configs to protect a multi-tenant environment is
> > > >
> > > > required.
> > > >
> > > > In a
> > > >
> > > > non-secure environment Authorizer is a blunt instrument either you
> > > >
> > > > end
> > > >
> > > > up
> > > >
> > > >
> > > > blocking everyone or allowing everyone.
> > > > I am asking to have server side that allow clients to create
> > > >
> > > >
> > > > topics or
> > > >
> > > > not
> > > >
> > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > replication-factor.
> > > >
> > > > -Harsha
> > > >
> > > > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com (
> > > >
> > > > ismaelj@gmail.com
> > > >
> > > > )
> > > >
> > > > wrote:
> > > >
> > > > Harsha,
> > > >
> > > > Rogue clients can use the admin client to create topics and
> > > >
> > > > partitions.
> > > >
> > > > ACLs and policies can help in that case as well as this one.
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha.
> > > >
> > > > io
> > > >
> > > > (
> > > >
> > > > kafka@harsha.io ) >
> > > >
> > > > wrote:
> > > >
> > > >
> > > > Hi Justine,
> > > > Thanks for the KIP.
> > > > "When server-side auto-creation is disabled, client-side
> > > >
> > > >
> > > > auto-creation
> > > >
> > > >
> > > > will try to use client-side configurations"
> > > > If I understand correctly, this KIP is removing any server-side
> > > >
> > > >
> > > > blocking
> > > >
> > > >
> > > > client auto creation of topic?
> > > > if so this will present potential issue of rogue client creating
> > > >
> > > >
> > > > ton of
> > > >
> > > > topic-partitions and potentially bringing down the service for
> > > >
> > > > everyone
> > > >
> > > > or
> > > >
> > > >
> > > > degrade the service itself.
> > > > By reading the KIP its not clear to me that there is a clear way to
> > > >
> > > >
> > > > block
> > > >
> > > > auto creation topics of all together from clients by server side
> > > >
> > > > config.
> > > >
> > > > Server side configs of default topic, partitions should take higher
> > > > precedence and client shouldn't be able to create a topic with
> > > >
> > > > higher
> > > >
> > > > no.of
> > > >
> > > > partitions, replication than what server config specifies.
> > > >
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > >
> > > > Hi all,
> > > > I made some changes to the KIP. Hopefully this configuration change
> > > >
> > > >
> > > > will
> > > >
> > > > make things a little clearer.
> > > >
> > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > > >
> > > > https://cwiki.
> > > >
> > > >
> > > > apache.org/confluence/display/KAFKA/ )
> > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > >
> > > >
> > > > Please let me know if you have any feedback or questions!
> > > >
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > >
> > > > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache.
> > > >
> > > > org (
> > > >
> > > > cmccabe@apache.org ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think you bring up a good point. It would be better if we didn't
> > > >
> > > > ever
> > > >
> > > > have to set up client-side configuration for this feature, and
> > > >
> > > > KIP-464
> > > >
> > > > would let us skip this entirely.
> > > >
> > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > >
> > > >
> > > > Hi Mickael,
> > > > I agree that KIP-464 works on newer brokers, but I was a bit
> > > >
> > > >
> > > > worried
> > > >
> > > > how
> > > >
> > > > things would play out on older brokers that* do not *have KIP 464
> > > >
> > > > included.
> > > >
> > > > Is it enough to throw an error in this case when producer configs
> > > >
> > > > are
> > > >
> > > > not
> > > >
> > > > specified?
> > > >
> > > > I think the right thing to do would be to log an error message in
> > > >
> > > > the
> > > >
> > > > client. We will need to have that capability in any case, to cover
> > > > scenarios like the client trying to auto-create a topic that they
> > > >
> > > > don't
> > > >
> > > > have permission to create. Or a client trying to create a topic on
> > > >
> > > > a
> > > >
> > > > broker
> > > >
> > > > so old that CreateTopicsRequest is not supported.
> > > >
> > > > The big downside to relying on KIP-464 is that it is a very recent
> > > >
> > > > feature
> > > >
> > > > -- so recent that it hasn't even made its way to any official
> > > >
> > > > Apache
> > > >
> > > > release. It's scheduled for the upcoming 2.4 release in a few
> > > >
> > > > months.
> > > >
> > > > So if you view this KIP as a step towards removing broker-side
> > > > auto-create, you might want to support older brokers just to
> > > >
> > > > accelerate
> > > >
> > > > adoption, and hasten the day when we can finally flip broker-side
> > > > auto-create to off (or even remove it entirely).
> > > >
> > > > I have to agree, though, that having client-side configurations for
> > > >
> > > > number
> > > >
> > > > of partitions and replication factor is messy. Maybe it would be
> > > >
> > > > worth
> > > >
> > > > it
> > > >
> > > > to restrict support to post-KIP-464 brokers, if we could avoid
> > > >
> > > > creating
> > > >
> > > > more configs.
> > > >
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> > > >
> > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > We can rely on KIP-464 which allows to omit the partition count or
> > > > replication factor when creating a topic. In that case, the broker
> > > >
> > > > defaults
> > > >
> > > > are used.
> > > >
> > > > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > >
> > > > Michael,
> > > > That makes sense to me!
> > > > To clarify, in the current state of the KIP, the producer does not
> > > >
> > > >
> > > > rely
> > > >
> > > > on
> > > >
> > > > the broker to autocreate--if the broker's config is disabled, then
> > > >
> > > > the
> > > >
> > > > producer can autocreate on its own with a create topic request (the
> > > >
> > > > same
> > > >
> > > >
> > > > type of request the admin client uses).
> > > > However, if both configs are enabled, the broker will autocreate
> > > >
> > > >
> > > > through
> > > >
> > > > a
> > > >
> > > > metadata request before the producer gets a chance. Of course, the
> > > >
> > > > way
> > > >
> > > > to
> > > >
> > > > avoid this, is to do as you suggested, and set
> > > >
> > > > the
> > > >
> > > > "allow_auto_topic_creation" field to false.
> > > >
> > > > I think the only thing we need to be careful with in this setup is
> > > >
> > > > without
> > > >
> > > > KIP 464, we can not use broker defaults for this topic. A user
> > > >
> > > > needs
> > > >
> > > > to
> > > >
> > > > specify the number of partition and replication factor in the
> > > >
> > > > config.
> > > >
> > > > An
> > > >
> > > > alternative to this is to have coded defaults for when these
> > > >
> > > > configs
> > > >
> > > > are
> > > >
> > > > unspecified, but it is not immediately apparent what these defaults
> > > >
> > > > should
> > > >
> > > > be.
> > > >
> > > >
> > > > Thanks again for reading my KIP,
> > > > Justine
> > > >
> > > >
> > > > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> > > >
> > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > >
> > > > Thanks for the response!
> > > > In my opinion, it would be better if the producer did not rely at
> > > >
> > > >
> > > > all
> > > >
> > > >
> > > > on the broker auto create feature as this is what we're aiming to
> > > > deprecate. When requesting metadata we can set the
> > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > creation. Then if the topic is not existing, send a
> > > >
> > > >
> > > > CreateTopicRequest.
> > > >
> > > > What do you think?
> > > >
> > > > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Currently the way it is implemented, the broker auto-creation
> > > >
> > > > configuration
> > > >
> > > > takes precedence. The producer will not use the CreateTopics
> > > >
> > > > request.
> > > >
> > > > (Technically it can--but the topic will already be created
> > > >
> > > > through
> > > >
> > > > the
> > > >
> > > > broker, so it will never try to create the topic.) It is possible
> > > >
> > > > to
> > > >
> > > > change this however, and I'd be happy to
> > > >
> > > > discuss
> > > >
> > > > the
> > > >
> > > > benefits of this alternative.
> > > >
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > >
> > > > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> > > >
> > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > In case auto creation is enabled on both the client and server,
> > > >
> > > > will
> > > >
> > > > the producer still use the AdminClient (CreateTopics request)
> > > >
> > > > to
> > > >
> > > > create topics? and not rely on the broker auto create. I'm
> > > >
> > > > guessing the
> > > >
> > > > answer is yes but can you make it explicit.
> > > >
> > > > Thank you
> > > >
> > > > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > >
> > > > Hi,
> > > > Just a friendly reminder to take a look at this KIP if you
> > > >
> > > >
> > > > have
> > > >
> > > > the
> > > >
> > > > time.
> > > >
> > > > I was thinking about broker vs. client default precedence,
> > > >
> > > > and I
> > > >
> > > > think it
> > > >
> > > > makes sense to keep the broker as the default used when both
> > > >
> > > > client-side
> > > >
> > > > and broker-side defaults are configured. The idea is that
> > > >
> > > > there
> > > >
> > > > would be
> > > >
> > > > pretty clear documentation, and that many systems with
> > > >
> > > > configurations
> > > >
> > > > that
> > > >
> > > > the client could not change would likely have the auto-create
> > > >
> > > > default
> > > >
> > > > off.
> > > >
> > > > (In cloud for example).
> > > >
> > > >
> > > > It also seems like in most cases, the consumer config
> > > > ' allow. auto. create. topics ( http://allow.auto.create.topics/
> > > >
> > > >
> > > > ) '
> > > >
> > > > was
> > > >
> > > > created to actually prevent
> > > >
> > > > the
> > > >
> > > > creation
> > > >
> > > > of
> > > >
> > > > topics, so the loss of creation functionality will not be a
> > > >
> > > > big
> > > >
> > > > problem.
> > > >
> > > > I'm happy to discuss any other compatibility problems or
> > > >
> > > > components
> > > >
> > > > of
> > > >
> > > > this KIP.
> > > >
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > >
> > > > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io )
> > > >
> > > > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I was looking at this KIP again, and there is a decision I
> > > >
> > > > made
> > > >
> > > > that I
> > > >
> > > > think is worth discussing.
> > > >
> > > >
> > > > In the case where both the broker and producer's
> > > > 'auto.create.topics.enable' are set to true, we have to
> > > >
> > > >
> > > > choose
> > > >
> > > > either
> > > >
> > > > the
> > > >
> > > > broker configs or the producer configs for the replication
> > > > factor/partitions.
> > > >
> > > > Currently, the decision is to have the broker defaults take
> > > >
> > > > precedence.
> > > >
> > > > (It is easier to do this in the implementation.) It also
> > > >
> > > > makes
> > > >
> > > > some
> > > >
> > > > sense
> > > >
> > > > for this behavior to take precedence since this behavior
> > > >
> > > > already
> > > >
> > > > occurs as
> > > >
> > > > the default.
> > > >
> > > > However, I was wondering if it would be odd for those who
> > > >
> > > > can
> > > >
> > > > only
> > > >
> > > > see
> > > >
> > > > the
> > > >
> > > > producer side to set configs for replication
> > > >
> > > > factor/partitions
> > > >
> > > > and
> > > >
> > > > see
> > > >
> > > > different results. Currently the documentation for the
> > > >
> > > > config
> > > >
> > > > states
> > > >
> > > > that
> > > >
> > > > the config values are only used when the broker config is
> > > >
> > > > not
> > > >
> > > > enabled,
> > > >
> > > > but
> > > >
> > > > this might not always be clear to the user. Changing the
> > > >
> > > > code
> > > >
> > > > to
> > > >
> > > > have
> > > >
> > > > the
> > > >
> > > > producer's configurations take precedence is possible, but
> > > >
> > > > I
> > > >
> > > > was
> > > >
> > > > wondering
> > > >
> > > > what everyone thought.
> > > >
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > >
> > > > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Just a quick update--
> > > >
> > > > It seems that enabling both the broker and producer
> > > >
> > > > configs
> > > >
> > > > works
> > > >
> > > > fine,
> > > >
> > > > except that the broker configurations for partitions,
> > > >
> > > > replication
> > > >
> > > > factor
> > > >
> > > >
> > > > take precedence.
> > > > I don't know if that is something we would want to
> > > >
> > > >
> > > > change, but
> > > >
> > > > I'll be
> > > >
> > > > updating the KIP for now to reflect this. Perhaps we would
> > > >
> > > > want to
> > > >
> > > > add more
> > > >
> > > > to the documentation of the the producer configs to
> > > >
> > > > clarify.
> > > >
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > >
> > > > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Colin,
> > > >
> > > > Thanks for looking at the KIP. I can definitely add to
> > > >
> > > > the
> > > >
> > > > title
> > > >
> > > > to
> > > >
> > > > make
> > > >
> > > > it more clear.
> > > >
> > > > It makes sense that both configurations could be turned
> > > >
> > > > on
> > > >
> > > > since
> > > >
> > > > there
> > > >
> > > > are many cases where the user can not control the
> > > >
> > > > server-side
> > > >
> > > > configurations. I was a little concerned about how both
> > > >
> > > > interacting
> > > >
> > > > would
> > > >
> > > > work out -- if there would be to many requests for new
> > > >
> > > > topics,
> > > >
> > > > for
> > > >
> > > > example.
> > > >
> > > > But it since it does make sense to allow both
> > > >
> > > > configurations
> > > >
> > > > enabled, I
> > > >
> > > > will test out some scenarios and I'll change the KIP to
> > > >
> > > > support
> > > >
> > > > this.
> > > >
> > > > I also agree with documentation about distinguishing the
> > > >
> > > > differences. I
> > > >
> > > > was having some trouble with the wording but I like the
> > > >
> > > > phrases
> > > >
> > > > "server-side" and "client-side." That's a good
> > > >
> > > > distinction I
> > > >
> > > > can
> > > >
> > > > use
> > > >
> > > > when
> > > >
> > > > describing.
> > > >
> > > > I'll try to update the KIP soon keeping everyone's input
> > > >
> > > > in
> > > >
> > > > mind.
> > > >
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > >
> > > > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> > > >
> > > > cmccabe@ apache. org ( cmccabe@apache.org )
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP. This seems like a good step towards
> > > >
> > > > removing
> > > >
> > > > server-side topic auto-creation.
> > > >
> > > > We should add included "client-side" to the title of
> > > >
> > > > the KIP
> > > >
> > > > somewhere,
> > > >
> > > > to make it clear that we're talking about client-side
> > > >
> > > > auto
> > > >
> > > > creation.
> > > >
> > > > The KIP says:
> > > >
> > > > In order to automatically create topics with the
> > > >
> > > > producer, the
> > > >
> > > > producer's
> > > >
> > > > auto.create.topics.enable config must be set to true
> > > >
> > > > and
> > > >
> > > > the
> > > >
> > > > broker
> > > >
> > > > config should be set to false
> > > >
> > > > From a user's point of view, this seems
> > > >
> > > > counter-intuitive.
> > > >
> > > > In
> > > >
> > > > order to
> > > >
> > > > auto-create topics the broker's
> > > >
> > > > auto.create.topics.enable
> > > >
> > > > config
> > > >
> > > > should be
> > > >
> > > > set to false? It seems like the server-side
> > > >
> > > > auto-create is
> > > >
> > > > unrelated to
> > > >
> > > > the client-side auto-create. We could have both turned
> > > >
> > > > on
> > > >
> > > > (and
> > > >
> > > > I'm
> > > >
> > > > sure
> > > >
> > > > that in the real world, people will try this
> > > >
> > > > configuration...)
> > > >
> > > > There's no
> > > >
> > > > reason not to support this, I think.
> > > >
> > > > We should add some documentation explaining the
> > > >
> > > > difference
> > > >
> > > > between
> > > >
> > > > server-side and client-side auto-creation. Without
> > > >
> > > > documentation,
> > > >
> > > > an admin
> > > >
> > > > might think that they had disabled all forms of
> > > >
> > > > auto-creation by
> > > >
> > > > setting
> > > >
> > > > the -side setting to false-- but this is not the case,
> > > >
> > > > of
> > > >
> > > > course.
> > > >
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > >
> > > > Hi Dhruvil,
> > > >
> > > >
> > > > Thanks for reading the KIP!
> > > > That was the general idea for deprecation. We would
> > > >
> > > >
> > > > log a
> > > >
> > > > warning
> > > >
> > > > when the
> > > >
> > > >
> > > > config is enabled on the broker.
> > > > I also believe that there would be a change to
> > > >
> > > >
> > > > documentation.
> > > >
> > > > If there is anything else that should be done, please
> > > >
> > > > let
> > > >
> > > > me
> > > >
> > > > know!
> > > >
> > > > Justine
> > > >
> > > > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> > > >
> > > > dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP, this is great!
> > > >
> > > > Could you add some more information about what
> > > >
> > > > deprecating
> > > >
> > > > the
> > > >
> > > > broker
> > > >
> > > > configuration means? Would we log a warning in the
> > > >
> > > > logs
> > > >
> > > > when
> > > >
> > > > auto
> > > >
> > > > topic
> > > >
> > > > creation is enabled on the broker, for example?
> > > >
> > > >
> > > > Thanks,
> > > > Dhruvil
> > > >
> > > >
> > > > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I'd like to start a discussion thread for KIP-487. This KIP plans
> > > >
> > > > to
> > > >
> > > > deprecate the current system of
> > > >
> > > > auto-creating
> > > >
> > > > topics
> > > >
> > > > through requests to the metadata and give the
> > > >
> > > > producer the
> > > >
> > > > ability to
> > > >
> > > > automatically create topics instead.
> > > >
> > > > More information can be found here:
> > > >
> > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > > >
> > > > https://cwiki.
> > > >
> > > >
> > > > apache.org/confluence/display/KAFKA/ )
> > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > >
> > > >
> > > >
> > > > Thank you,
> > > > Justine Olshan
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Boyang Chen <re...@gmail.com>.

Hey Justin and Jiamei,

I read the KIP and skimmed over the discussion. One thing I'm not fully
convinced of is why we need to deprecate the server side auto topic
creation logic, which seems orthogonal towards whether a client wants to
create the topic or not. Won't it be more natural to assume that only when
both server and client agree on turning on the switch, will a topic get
created?

Some clarifications would also be appreciated:

1. Could we include a link to KIP-464 and explain its relation to KIP-487?
It's very hard to read through the proposal when readers only have a
reference number to some KIP that is not briefed.

2. The KIP suggests, " In the producer, auto-creation of a topic will occur
through a specific request rather than through a side effect of requesting
metadata." Could we be specific such as whether we are going to introduce a
separate RPC, or just send another CreateTopicRequest?

Boyang

On Wed, Jun 17, 2020 at 8:51 AM jiamei xie <ji...@gmail.com> wrote:

> Hi all
> For
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> ,  It has not been updated for a long time. And I made some update, which
> has been pushed to https://github.com/apache/kafka/pull/8831
>
> MetadataRequest has method Builder(List<String> topics, boolean
> allowAutoTopicCreation) by which we can set whether to enable
> allowAutoTopicCreation from producer.
> By default, allowAutoTopicCreation on Producer is true. And only if when
> the allowAutoTopicCreation of Broker and Producer are true, the topic can
> be auto-created.
>
> Besides, the test cases are changed:
> There are 4 cases for brokerAutoTopicCreationEnable and
> producerAutoCreateTopicsPolicy, Check if the topic is created under these
> four cases.
>      If brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy
> are true:  assertTrue(topicCreated)
>      else : intercept[ExecutionException]
>
> Looking forward to your feedback and comments. Thanks.
>
> Best wishes
> Jiamei Xie
>
> On 2019/08/12 15:50:22, Harsha Chintalapani <ka...@harsha.io> wrote:
> > On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi all,
> > >
> > > A few points:
> > >
> > > 1. I think the way backwards compatibility is being used here is not
> > > correct. Any functionality that is only enabled if set via a config is
> > > backwards compatible. People may disagree with the functionality or the
> > > config, but it's not a backwards compatibility issue.
> > >
> >
> > We are talking about both broker and producer as a  single entity and run
> > by the same team/users. Allowing newer producer to create topics on a
> older
> > broker when auto.create.topics.enable set to false, breaks  server side
> > contract that this config offered from the beginning.  IMO, it clearly
> > isn't backward compatible. User who set auto.create.topic.enable on
> broker
> > will not be the same who will turn it on producer side .
> >
> >
> > > 2. It's an interesting question if auto topic creation via the producer
> > > should be a server driven choice or not. I can see the desire to have a
> > > server-driven default, but it seems like this is often application
> > > specific. Because the functionality is trivially available via
> AdminClient
> > > (released 2 years ago), it's not quite possible to control what
> > > applications do without the use of ACLs or policies today.
> > >
> > >
> > >
> > Producers & consumers are the majority of the clients in Kafka ecosystem.
> > Just because AdminClient shipped a while back that doesn't mean all users
> > adopting to it. To this day lot more users are aware of Producer &
> Consumer
> > APIs and running them in production compare to AdminClient.
> >
> >
> > > 3. Changing the create topics request in this way is highly
> unintuitive in
> > > my opinion and it relies on every client to pass the new field. For
> > > example, if librdkafka added auto create functionality by relying on
> their
> > > AdminClient, it would behave differently than what is proposed here.
> > > Forcing every client to implement this change when calling auto create
> from
> > > the producer specifically seems odd
> > >
> >
> > I am not sure why its unintuitive , protocols change. We add or upgrade
> the
> > existing protocols all the time.
> >
> >
> > Thanks,
> > Harsha
> >
> > .
> > >
> > > Ismael
> > >
> > > On Thu, Aug 8, 2019 at 11:51 AM Jun Rao <ju...@confluent.io> wrote:
> > >
> > > Hi, Justine,
> > >
> > > Thanks for the KIP. Overall, it seems to be a good improvement.
> > >
> > > However, I think Harsha's point seems reasonable. We had
> > > auto.create.topics.enable config on the broker to allow admins to
> disable
> > > topic creation from the producer/consumer clients before we had the
> > > security feature. The need for that config is reduced with the security
> > > feature, but may still be present since not all places have security
> > > enabled. It's true that a non-secured environment is vulnerable to some
> > > additional attacks, but producer/consumer are the most common way for a
> > > user to interact with the broker. So, keeping that config for backward
> > > compatibility could still be useful if it's not introducing too much
> effort
> > > or extra confusion.
> > >
> > >
> > > Here is a one potential alternative way that I was thinking. We add a
> new
> > > field in the CreateTopicRequest to indicate whether it's from the
> producer
> > > or not. If auto.create.topics.enable is false, CreateTopicRequest from
> the
> > > producer will be rejected. We probably don't need to introduce the new
> > > config (which seems a bit hard to explain) in the producer. Instead,
> the
> > > new producer always uses MetadataRequest with AllowAutoTopicCreation
> set to
> > > false to get the metadata and if the metadata is not present, send the
> new
> > > CreateTopicRequest
> > > (assuming the broker supports it) to try to create the topic
> > > automatically. Whether the creation is allowed or not will be
> determined by
> > > the broker. This will make the behavior backward compatible and we can
> > > still achieve the main goal of the KIP, which is not relying on
> > > MetadataRequest for topic creation. What do you think?
> > >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Aug 8, 2019 at 1:34 AM M. Manna <ma...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > >
> > > If I may, perhaps you could simplify everything by using only
> > > 'auto.create.topics.enable' as a value along with true. In other words,
> > >
> > >
> > > the
> > >
> > > public interfaces section should only have
> > >
> > > [true,auto.create.topics.enable,
> > >
> > > false].
> > >
> > > The reason for this is that auto.create.topics.enable is already known
> to
> > > users as a "Server-SIde" config. So all you are saying is
> > >
> > > a) To avoid day 1 impact, it will follow whatever
> > >
> > > auto.create.topics.enable
> > >
> > >
> > > value is set.
> > > b) False means - no client side topic creation
> > > c) True means client side topic creation.
> > >
> > >
> > > It saves creating 2 more new strings :). But not too expensive anyway.
> > >
> > > Also, when you deprecate auto.create.topics.enable - you must provide
> > > sufficient logic to ensure that things like rolling upgrade doesn't
> > > temporarily break anything. I apologise if you have already accounted
> for
> > > this, but wanted to mention since I didn't notice this on the KIP.
> > >
> > > Let me know how this sounds.
> > >
> > > Regards,
> > >
> > > On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jo...@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Harsha,
> > >
> > > I think my message may have gotten lost in all the others.
> > >
> > > Two of the goals of this KIP are to 1) allow auto-creation on specific
> > > clients when the broker default is false and 2) eventually replace the
> > > broker config.
> > >
> > > In order to accomplish these two goals, we need the producer to be able
> > >
> > > to
> > >
> > >
> > > create topics despite the broker config. (How can we replace a function
> > > when we rely on it?)
> > > I think at this point we have a fundamental disagreement in what we
> > >
> > >
> > > should
> > >
> > >
> > > allow the producer to do.
> > > In my previous message I mentioned a config that would allow for the
> > >
> > >
> > > broker
> > >
> > > to prevent producer auto-creation. (It would be disabled by default.)
> > >
> > > It
> > >
> > > would fix your issue for now, but could lead to more complications
> > >
> > > later.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz>
> > >
> > > wrote:
> > >
> > > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> > >
> > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe <
> > >
> > > cmccabe@apache.org
> > >
> > > wrote:
> > >
> > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> > >
> > >
> > > Hi Colin,
> > > "Hmm... I'm not sure I follow. Users don't have to build their own
> > > tooling, right? They can use any of the shell scripts that we've
> > >
> > >
> > > shipped
> > >
> > > in
> > >
> > > the last few releases. For example, if any of your users run it,
> > >
> > > this
> > >
> > > shell
> > >
> > > script will delete all of the topics from your non-security-enabled
> > > cluster:
> > >
> > >
> > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --delete
> > > --topic
> > >
> > >
> > > They will need to fill in the correct bootstrap servers list, of
> > >
> > > course,
> > >
> > > not localhost. This deletion script will work on some pretty old
> > >
> > > brokers,
> > >
> > > even back to the 0.10 releases. It seems a little odd to trust your
> > >
> > > users
> > >
> > > with this power, but not trust them to avoid changing a particular
> > > configuration key."
> > >
> > > The above will blocked by the server if we set delete.topic.enable
> > >
> > > to
> > >
> > > false and thats exactly what I am asking for.
> > >
> > > Hi Harsha,
> > >
> > > I was wondering if someone was going to bring up that configuration
> > >
> > > :)
> > >
> > > it's an interesting complication, but globally disabling topic
> > >
> > > deletion
> > >
> > > is
> > >
> > > not very practical for most use-cases.
> > >
> > > In any case, there are plenty of other bad things that users with
> > >
> > > full
> > >
> > > permissions can do that aren't blocked by any server configuration.
> > >
> > > For
> > >
> > > example, they can delete every record in every topic. I can write a
> > >
> > > script
> > >
> > > for that too, and there's no server configuration you can set to
> > >
> > > disable
> > >
> > > it. Or I could simply create hundreds of thousands of topics, until
> > >
> > > cluster
> > >
> > > performance becomes unacceptable (this will be even more of a
> > >
> > > problem
> > >
> > > if
> > >
> > > someone configured delete.topic.enable as false). Or publish bad
> > >
> > > data
> > >
> > > to
> > >
> > > every topic, etc. etc.
> > >
> > > The point I'm trying to make here is that you can't rely on these
> > >
> > > kind
> > >
> > > of
> > >
> > > server-side configurations for security. At most, they're a way to
> > >
> > > set
> > >
> > > up
> > >
> > > certain very simple policies. But the policies are so simple that
> > >
> > > they're
> > >
> > > hardly ever useful any more.
> > >
> > > For example, if the problem you want to solve is that you want a
> > >
> > > user
> > >
> > > to
> > >
> > > only be able to create 50 topics and not delete anyone else's
> > >
> > > topics,
> > >
> > > you
> > >
> > > can solve that with a CreateTopicsPolicy that limits the number of
> > >
> > > topics,
> > >
> > > and some ACLs. There's no combination of auto.create.topics.enable
> > >
> > > and
> > >
> > > delete.topic.enable that will help here.
> > >
> > > Hi Colin,
> > >
> > > Well you gave the example that a user can delete the
> > >
> > > topics
> > >
> > > just by running that script :).
> > >
> > > I understand there are open APIs in Kafka and can lead to rogue
> > >
> > > clients
> > >
> > > taking advantage of it without proper security in place.
> > >
> > > What I am asking so far in this thread is , this KIP is changing
> > >
> > > the
> > >
> > > producer behavior and its not backward compatible.
> > >
> > > The change is backwards compatible. The default will still be
> > >
> > > server-side
> > >
> > > topic auto-creation, just like now.
> > >
> > > You will have to specifically change the producer config to get the
> > >
> > > new
> > >
> > > behavior.
> > >
> > > I disagree. Today server can turn off the topic creation neither
> > >
> > > producer
> > >
> > > or consumer can create a topic. With this KIP , producer can create a
> > >
> > > topic
> > >
> > > by turning on client side config when server side config is turned
> > >
> > > off.
> > >
> > > We can still achieve
> > >
> > > the main goal of the KIP which is to change MetadataRequest
> > >
> > > creating
> > >
> > > topics and send a CreateTopicRequest from Producer and also keep
> > >
> > > the
> > >
> > > server
> > >
> > > side config to have precedence. This KIP originally written to
> > >
> > > have
> > >
> > > server
> > >
> > > side preference and there is not much explanation why it changed to
> > >
> > > have
> > >
> > > Producer side preference.
> > >
> > > Arguing that AdminClient can do that and so we are going to make
> > >
> > > Producer
> > >
> > > do the same doesn't make sense.
> > >
> > > "The downside is that if we wanted to check a server side
> > >
> > > configuration
> > >
> > > before sending the create topics request, the code would be more
> > >
> > > complex.
> > >
> > > The behavior would also not be consistent with how topic
> > >
> > > auto-creation
> > >
> > > is
> > >
> > > handled in Kafka Streams."
> > >
> > > I am not sure why you need to check server side configuration
> > >
> > > before
> > >
> > > sending create topics request. A user enables producer side config
> > >
> > > to
> > >
> > >
> > > create topics.
> > > Producer sends a request to the broker and if the broker has
> > > auto.topic.create.enable to true (default) it will allow creation
> > >
> > >
> > > of
> > >
> > > topics. If it set to false it returns error back to the client.
> > >
> > > auto.topic.create.enable has never affected CreateTopicsRequest. If
> > >
> > > you
> > >
> > > submit a CreateTopicsRequest and you are authorized, a topic will
> > >
> > > be
> > >
> > > created, regardless of what the value of auto.topic.create.enable
> > >
> > > is.
> > >
> > > This
> > >
> > > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I
> > >
> > > think?)
> > >
> > > I don't see how this behavior will be different in Kafka streams.
> > >
> > > By
> > >
> > > default server allows the topic creation and with this KIP, It will
> > >
> > > only
> > >
> > > allow creation of topic when both producer and server side are
> > >
> > > turned
> > >
> > > on.
> > >
> > > Its exactly the same behavior in KIP-361.
> > >
> > > I suggest running a streams job as a test. It creates the topics it
> > >
> > > needs
> > >
> > > using AdminClient. The value of auto.topic.create.enable is not
> > >
> > > relevant.
> > >
> > > Whether it is true or false, the topics will still be created.
> > >
> > > "In general, it would be nice if we could keep the security and
> > >
> > > access
> > >
> > > control model simple and not introduce a lot of special cases and
> > > exceptions. Kafka has basically converged on using ACLs and
> > > CreateTopicPolicy classes to control who can create topics and
> > >
> > > where.
> > >
> > > Adding more knobs seems like a step backwards, especially when the
> > >
> > > proposed
> > >
> > > knobs don't work consistently across components, and don't provide
> > >
> > > true
> > >
> > > security." This is not about access control at all. Shipping sane
> > >
> > > defaults
> > >
> > > should be prioritized.
> > >
> > > I don't think the default is really the question here. I think
> > >
> > > everyone
> > >
> > > agrees that the default for client-side auto-topic creation should
> > >
> > > be
> > >
> > > off.
> > >
> > > My point goes back to KIP-361 why we kept the server side config to
> > >
> > > have
> > >
> > > preference. We can keep bringing back the discussion of security
> > >
> > > policies
> > >
> > > but a producer client overiding server side setting is not just
> > >
> > > security
> > >
> > > policy issue .
> > >
> > > "Producer client can override server side setting and not consumer
> > >
> > > and
> > >
> > > also producer can only override creation of topic but no the
> > >
> > > replication
> > >
> > > and partitions. " This doesn't make sense to me and why we want to
> > > introduce such a confusing behavior.
> > >
> > > The scenarios that you're describing (such as dealing with a poorly
> > > configured client that tries to create topics it should not) do
> > >
> > > seem
> > >
> > > to
> > >
> > > be
> > >
> > > about access control.
> > >
> > > We keep talking about CreateTopicPolicy and yet we don't have
> > >
> > > default
> > >
> > > one
> > >
> > > and asking every user of Kafka implement their own doesn't make
> > >
> > > sense
> > >
> > > here.
> > >
> > > The point of CreateTopicPolicy is exactly that you can implement
> > >
> > > your
> > >
> > > own,
> > >
> > > though. It's a way of customizing what the policy is in your
> > >
> > > specific
> > >
> > > cluster.
> > >
> > > I agree that most people don't want to write a custom policy. But
> > >
> > > most
> > >
> > > of
> > >
> > > those people are satisfied with just setting up ACLs to set up
> > >
> > > simple
> > >
> > > policies like this user can create topics, that user can't, etc.
> > >
> > > That's
> > >
> > > why
> > >
> > > I suggested adding support for ACLs to insecure clusters might be
> > >
> > > helpful.
> > >
> > > Also, just as a side note, wouldn't it would be impossible for us
> > >
> > > to
> > >
> > > specify a default CreateTopicsPolicy that did anything at this
> > >
> > > point
> > >
> > > anyway
> > >
> > > without breaking backwards compatibility? Maybe I'm
> > >
> > > misunderstanding
> > >
> > > the
> > >
> > > proposal.
> > >
> > > I am asking about exact behavior that we shipped for consumer side
> > >
> > > https:/
> > >
> > > / cwiki. apache. org/ confluence/ display/ KAFKA/
> > >
> > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > >
> > >
> > > (
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > >
> > >
> > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > >
> > > )
> > >
> > > I still didn't get any response why this behavior shouldn't be
> > >
> > > exactly
> > >
> > > like Kafka consumer and why do we want to have producer to
> > >
> > > overrider
> > >
> > > server
> > >
> > > side config and while not allowing consumer to do so. We are not
> > >
> > > even
> > >
> > > allowing the same contract and this will cause more confusion from
> > >
> > > the
> > >
> > > users standpoint.
> > >
> > > Hmm. The consumer already has a way to override the server side
> > > configuration. See KIP-361: Add Consumer Configuration to Disable
> > >
> > > Auto
> > >
> > > Topic Creation. This lets the consumer skip auto-creating topics,
> > >
> > > even
> > >
> > > if
> > >
> > > auto-creation is enabled on the broker.
> > >
> > > Yes that's what I am saying and it doesn't allow topic creation if
> > >
> > > the
> > >
> > > server side auto-creation is disabled and in consumer its enabled.
> > >
> > > I
> > >
> > > would like to see the exact behavior for producer as stated in
> > >
> > > KIP-361.
> > >
> > > To be fair, the KIP should probably discuss why we don't want to
> > >
> > > implement
> > >
> > > client-side topic creation in the consumer in "rejected
> > >
> > > alternatives."
> > >
> > > Maybe Justine can add more context here in the KIP.
> > >
> > > The last time we talked about this, the reasoning was that we
> > >
> > > wanted
> > >
> > > to
> > >
> > > eventually get rid of consumer-side auto-topic creation entirely,
> > >
> > > and
> > >
> > > so
> > >
> > > it
> > >
> > > wasn't worth implementing an improved way of doing it. Producer
> > >
> > > side
> > >
> > > auto
> > >
> > > topic-creation, in contrast, will probably stick around, although
> > >
> > > we'd
> > >
> > > like
> > >
> > > to deprecate and remove the problematic server-side mechanism over
> > >
> > > time.
> > >
> > > We should implement the producer side topic creation with proper
> > >
> > > create
> > >
> > > topic request and I am in favor of this KIP.
> > >
> > > All I am asking is don't make the producer to override server side
> > >
> > > config
> > >
> > > and keep it similar to KIP-361 just like consumer, it honors what
> > >
> > > set
> > >
> > > on
> > >
> > > server side which takes the precedence. Changing this behavior in
> producer
> > > is backward incompatible and will catch users by surprise.
> > >
> > > All arguments for enforcing producer side overriding config can
> > >
> > > still
> > >
> > > be
> > >
> > > achieved. Server side auto creation turned on by default and any
> > >
> > > new
> > >
> > > producer client setting their auto creation config to true will
> > >
> > > create
> > >
> > > topics in default behavior and any user who set server side to
> > >
> > > false
> > >
> > > and
> > >
> > > upgrades to latest kafka with this KIP will not see any unwanted
> > >
> > > behavior
> > >
> > > from clients. IMO this is not a unreasonable request on this KIP
> > >
> > > and
> > >
> > > this
> > >
> > > requested behavior is exactly what the KIP initially proposed.
> > >
> > > Thanks,
> > >
> > > Harsha
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache.
> > >
> > > org (
> > >
> > > cmccabe@apache.org ) > wrote:
> > >
> > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > >
> > > Not sure how the AdminClient applies here, It is an external API
> > >
> > > and
> > >
> > > is
> > >
> > > not part of KafkaProducer so any user who updates to latest version
> > >
> > > of
> > >
> > >
> > > Kafka with this feature get to create the topics.
> > > They have to build a tooling around AdminClient allow themselves to
> > >
> > >
> > > create
> > >
> > > topics.
> > >
> > > Hi Harsha,
> > >
> > > Hmm... I'm not sure I follow. Users don't have to build their own
> > >
> > > tooling,
> > >
> > > right? They can use any of the shell scripts that we've shipped in
> > >
> > > the
> > >
> > > last
> > >
> > > few releases. For example, if any of your users run it, this shell
> > >
> > > script
> > >
> > > will delete all of the topics from your non-security-enabled
> > >
> > > cluster:
> > >
> > >
> > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --delete
> > > --topic
> > >
> > >
> > > They will need to fill in the correct bootstrap servers list, of
> > >
> > > course,
> > >
> > > not localhost. This deletion script will work on some pretty old
> > >
> > > brokers,
> > >
> > > even back to the 0.10 releases. It seems a little odd to trust your
> > >
> > > users
> > >
> > > with this power, but not trust them to avoid changing a particular
> > > configuration key.
> > >
> > > There is no behavior in Kafka producer that allowed users to delete
> > >
> > > the
> > >
> > > topics or delete the records. So citing them as an example doesn't
> > >
> > > makes
> > >
> > > sense in this context.
> > >
> > > I think Kafka Streams is relevant here. After all, it's software
> > >
> > > that
> > >
> > > we
> > >
> > > ship as part of the official Kafka release. And it auto-creates
> > >
> > > topics
> > >
> > > even
> > >
> > > when auto.create.topics.enable is set to false on the broker.
> > >
> > > I think that auto.create.topics.enable was never intended as a
> > >
> > > security
> > >
> > > setting to control access. It was intended as a way to disable the
> > > broker-side auto-create feature specifically, because there were
> > >
> > > some
> > >
> > > problems with that specific feature. Broker-side auto-creation is
> > > frustrating because it's on by default, and because it can
> > >
> > > auto-create
> > >
> > > topics even if the producer or consumer didn't explicitly ask for
> > >
> > > that.
> > >
> > > Neither of those problems applies to this KIP: producers have to
> > > specifically opt in, and it won't be on by default. Basically, we
> > >
> > > think
> > >
> > > that client-side auto-creation is actually a lot better-- hence
> > >
> > > this
> > >
> > > KIP
> > >
> > > :)
> > >
> > >
> > > But there is
> > > a functionality which allowed creation of topics if they don't
> > >
> > >
> > > exist
> > >
> > > in
> > >
> > > the
> > >
> > > cluster and this behavior could be controlled by a config on the
> > >
> > > server
> > >
> > > side. Now with this KIP we are allowing producer to make that
> > >
> > > decision
> > >
> > > without any gateway on the server via configs. Any user who is not
> > >
> > > aware
> > >
> > > of
> > >
> > >
> > > such changes
> > > can accidentally create these topics and we are essentially
> > >
> > >
> > > removing
> > >
> > > a
> > >
> > > config that exists in brokers today to block this accidental
> > >
> > > creation
> > >
> > > and
> > >
> > > allowing clients to take control.
> > >
> > > Again, I hope I'm not misinterpreting, but I don't see how this can
> > >
> > > be
> > >
> > > turned on accidentally. The user would have to specifically turn
> > >
> > > this
> > >
> > > on
> > >
> > > in
> > >
> > > the producer by setting the configuration key.
> > >
> > > I still didn't get any positives of not having server side configs?
> > >
> > > if
> > >
> > > you
> > >
> > > want to turn them on and allow any client to create topics set the
> > >
> > > default
> > >
> > >
> > > to true
> > > and allow users who doesn't want to have this feature let them turn
> > >
> > >
> > > them
> > >
> > > off. It will be the exact behavior as it is today , as far as
> > >
> > > producer
> > >
> > > is
> > >
> > >
> > > concerned. I am not
> > > understanding why not having server side configs to gateways such a
> > >
> > >
> > > hard
> > >
> > > requirement and this behavior exists today. As far I am concerned
> > >
> > > this
> > >
> > > breaks the backward compatibility.
> > >
> > > The downside is that if we wanted to check a server side
> > >
> > > configuration
> > >
> > > before sending the create topics request, the code would be more
> > >
> > > complex.
> > >
> > > The behavior would also not be consistent with how topic
> > >
> > > auto-creation
> > >
> > > is
> > >
> > > handled in Kafka Streams.
> > >
> > > In general, it would be nice if we could keep the security and
> > >
> > > access
> > >
> > > control model simple and not introduce a lot of special cases and
> > > exceptions. Kafka has basically converged on using ACLs and
> > > CreateTopicPolicy classes to control who can create topics and
> > >
> > > where.
> > >
> > > Adding more knobs seems like a step backwards, especially when the
> > >
> > > proposed
> > >
> > > knobs don't work consistently across components, and don't provide
> > >
> > > true
> > >
> > > security.
> > >
> > > Maybe we should support an insecure mode where there are still
> > >
> > > users
> > >
> > > and
> > >
> > > ACLs. That would help people who don't want to set up Kerberos,
> > >
> > > etc.
> > >
> > > but
> > >
> > > still want to protect against misconfigured clients or accidents.
> > >
> > > Hadoop
> > >
> > > has something like this, and I think it was useful. NFS also
> > >
> > > supported
> > >
> > > (supports?) a mode where you just pass whatever user ID you want
> > >
> > > and
> > >
> > > the
> > >
> > > system believes you. These things clearly don't protect against
> > >
> > > malicious
> > >
> > > users, but they can help set up policies when needed.
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache.
> > >
> > > org (
> > >
> > > cmccabe@apache.org ) > wrote:
> > >
> > > Hi Harsha,
> > >
> > > Thanks for taking a look.
> > >
> > > I'm not sure I follow the discussion about AdminClient.
> > >
> > > KafkaAdminClient
> > >
> > > has been around for about 2 years at this point as a public class.
> > >
> > > There
> > >
> > > are many programs that use it to automatically create topics. Kafka
> > > Streams does this, for example. If any of your users run Kafka
> > >
> > > Streams,
> > >
> > > they will be auto-creating topics, regardless of what setting you
> > >
> > > use
> > >
> > > for
> > >
> > > auto.create.topics.enable.
> > >
> > > A big part of the annoyance of auto-topic creation right now is
> > >
> > > that
> > >
> > > it is
> > >
> > > on by default. The new configuration proposed by KIP-487 wouldn't
> > >
> > > be.
> > >
> > > Users would have to explicitly opt in to the new behavior of
> > >
> > > client-side
> > >
> > > topic creation. If you run without security, you're already
> > >
> > > putting a
> > >
> > > huge
> > >
> > > amount of trust in your users. For example, you trust them not to
> > >
> > > delete
> > >
> > > records with the kafka-delete-records. sh (
> > >
> > > http://kafka-delete-records.
> > >
> > >
> > > sh/
> > > ) command, or delete topics with kafka-topics. sh (
> > >
> > >
> > > http://kafka-topics.
> > >
> > >
> > > sh/
> > > ). Trusting them not to set a certain config value seems minor in
> > > comparison, right?
> > >
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > >
> > >
> > > Hi,
> > > Even with policies one needs to implement that, so for every
> > >
> > >
> > > user who
> > >
> > > doesn't want a producer to create topics or have limits around
> > >
> > > partitions &
> > >
> > > replication factor they have to implement a policy. The KIP is
> > >
> > > changing
> > >
> > > the behavior , it might not be introducing
> > >
> > > the
> > >
> > > new functionality but it will enable producers to override the
> > >
> > > create
> > >
> > > topic
> > >
> > > config settings on the broker. What I am asking for to provide a
> > >
> > > config
> > >
> > > that will disable auto creation of topics and if its enabled set
> > >
> > > some
> > >
> > > sane
> > >
> > > defaults so that clients can create a topic with in those limits. I
> > >
> > > don't
> > >
> > >
> > > see how this not related to this KIP.
> > > If the server config options as I suggested doesn't interest
> > >
> > >
> > > you at
> > >
> > >
> > > least have a default CreateTopicPolices in place.
> > > To give an example, In our environment we disable the
> > > auto.create.topic.enable and force users to go through a
> > >
> > >
> > > centralized
> > >
> > > service as we want capture more details about the topic creation
> > >
> > > and
> > >
> > > requirements. With this KIP, a producer can create a topic with no
> > >
> > > bounds.
> > >
> > > Another example max.message.size we define that at cluster level
> > >
> > > and one
> > >
> > > can override max.messsage.size at topic level, any misconfiguration
> > >
> > > at
> > >
> > > this
> > >
> > > will cause service degradation. Its not always about the rogue
> > >
> > > clients,
> > >
> > > users can easily misconfigure and can cause an outage. Again we can
> > >
> > > talk
> > >
> > > about CreateTopicPolicy but without having a
> > >
> > > default
> > >
> > > implementation and asking everyone to implement their own while
> > >
> > > changing
> > >
> > > the behavior in producer doesn't make sense to me.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me.
> > >
> > > uk (
> > >
> > > ismael@juma.me.uk ) >
> > >
> > > wrote:
> > >
> > > Hi Harsha,
> > >
> > > I mentioned policies and the authorizer. For example, with
> > > CreateTopicPolicy, you can implement the limits you describe. If
> > >
> > > you
> > >
> > > have
> > >
> > > ideas of how that should be improved, please submit a KIP. My
> > >
> > > point is
> > >
> > > that
> > >
> > > this KIP is not introducing any new functionality with regards to
> > >
> > > what
> > >
> > > rogue clients can do. It's using the existing protocol that is
> > >
> > > already
> > >
> > > exposed via the AdminClient. So, I don't think we need to address
> > >
> > > it in
> > >
> > > this KIP. Does that make sense?
> > >
> > > Ismael
> > >
> > > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> > >
> > > kafka@ harsha. io ( kafka@harsha.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Ismael,
> > > Sure AdminClient can do that and we should've shipped a config or
> > >
> > >
> > > use
> > >
> > > the
> > >
> > > existing one to block that. Not all users are yet to upgrade to
> > >
> > > AdminClient
> > >
> > > and start using that to cause issues yet. In shared environment we
> > >
> > > should
> > >
> > > allow server to set sane defaults and not allow every client to go
> > >
> > > ahead
> > >
> > > create random no.of topic/partitions and replication factor. Even
> > >
> > > if
> > >
> > > the
> > >
> > > users want to allow topic creation proposed in the KIP , it makes
> > >
> > > sense to
> > >
> > > have some guards against the no.of partitions and replication
> > >
> > > factor.
> > >
> > > Authorizer is not always an answer to block requests and having
> > >
> > > users
> > >
> > > set
> > >
> > > server side configs to protect a multi-tenant environment is
> > >
> > > required.
> > >
> > > In a
> > >
> > > non-secure environment Authorizer is a blunt instrument either you
> > >
> > > end
> > >
> > > up
> > >
> > >
> > > blocking everyone or allowing everyone.
> > > I am asking to have server side that allow clients to create
> > >
> > >
> > > topics or
> > >
> > > not
> > >
> > > , if they are allowed set a ceiling on max no.of partitions and
> > > replication-factor.
> > >
> > > -Harsha
> > >
> > > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com (
> > >
> > > ismaelj@gmail.com
> > >
> > > )
> > >
> > > wrote:
> > >
> > > Harsha,
> > >
> > > Rogue clients can use the admin client to create topics and
> > >
> > > partitions.
> > >
> > > ACLs and policies can help in that case as well as this one.
> > >
> > > Ismael
> > >
> > > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha.
> > >
> > > io
> > >
> > > (
> > >
> > > kafka@harsha.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Hi Justine,
> > > Thanks for the KIP.
> > > "When server-side auto-creation is disabled, client-side
> > >
> > >
> > > auto-creation
> > >
> > >
> > > will try to use client-side configurations"
> > > If I understand correctly, this KIP is removing any server-side
> > >
> > >
> > > blocking
> > >
> > >
> > > client auto creation of topic?
> > > if so this will present potential issue of rogue client creating
> > >
> > >
> > > ton of
> > >
> > > topic-partitions and potentially bringing down the service for
> > >
> > > everyone
> > >
> > > or
> > >
> > >
> > > degrade the service itself.
> > > By reading the KIP its not clear to me that there is a clear way to
> > >
> > >
> > > block
> > >
> > > auto creation topics of all together from clients by server side
> > >
> > > config.
> > >
> > > Server side configs of default topic, partitions should take higher
> > > precedence and client shouldn't be able to create a topic with
> > >
> > > higher
> > >
> > > no.of
> > >
> > > partitions, replication than what server config specifies.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Hi all,
> > > I made some changes to the KIP. Hopefully this configuration change
> > >
> > >
> > > will
> > >
> > > make things a little clearer.
> > >
> > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > >
> > > https://cwiki.
> > >
> > >
> > > apache.org/confluence/display/KAFKA/ )
> > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > >
> > >
> > > Please let me know if you have any feedback or questions!
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache.
> > >
> > > org (
> > >
> > > cmccabe@apache.org ) >
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think you bring up a good point. It would be better if we didn't
> > >
> > > ever
> > >
> > > have to set up client-side configuration for this feature, and
> > >
> > > KIP-464
> > >
> > > would let us skip this entirely.
> > >
> > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > >
> > >
> > > Hi Mickael,
> > > I agree that KIP-464 works on newer brokers, but I was a bit
> > >
> > >
> > > worried
> > >
> > > how
> > >
> > > things would play out on older brokers that* do not *have KIP 464
> > >
> > > included.
> > >
> > > Is it enough to throw an error in this case when producer configs
> > >
> > > are
> > >
> > > not
> > >
> > > specified?
> > >
> > > I think the right thing to do would be to log an error message in
> > >
> > > the
> > >
> > > client. We will need to have that capability in any case, to cover
> > > scenarios like the client trying to auto-create a topic that they
> > >
> > > don't
> > >
> > > have permission to create. Or a client trying to create a topic on
> > >
> > > a
> > >
> > > broker
> > >
> > > so old that CreateTopicsRequest is not supported.
> > >
> > > The big downside to relying on KIP-464 is that it is a very recent
> > >
> > > feature
> > >
> > > -- so recent that it hasn't even made its way to any official
> > >
> > > Apache
> > >
> > > release. It's scheduled for the upcoming 2.4 release in a few
> > >
> > > months.
> > >
> > > So if you view this KIP as a step towards removing broker-side
> > > auto-create, you might want to support older brokers just to
> > >
> > > accelerate
> > >
> > > adoption, and hasten the day when we can finally flip broker-side
> > > auto-create to off (or even remove it entirely).
> > >
> > > I have to agree, though, that having client-side configurations for
> > >
> > > number
> > >
> > > of partitions and replication factor is messy. Maybe it would be
> > >
> > > worth
> > >
> > > it
> > >
> > > to restrict support to post-KIP-464 brokers, if we could avoid
> > >
> > > creating
> > >
> > > more configs.
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> > >
> > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > We can rely on KIP-464 which allows to omit the partition count or
> > > replication factor when creating a topic. In that case, the broker
> > >
> > > defaults
> > >
> > > are used.
> > >
> > > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Michael,
> > > That makes sense to me!
> > > To clarify, in the current state of the KIP, the producer does not
> > >
> > >
> > > rely
> > >
> > > on
> > >
> > > the broker to autocreate--if the broker's config is disabled, then
> > >
> > > the
> > >
> > > producer can autocreate on its own with a create topic request (the
> > >
> > > same
> > >
> > >
> > > type of request the admin client uses).
> > > However, if both configs are enabled, the broker will autocreate
> > >
> > >
> > > through
> > >
> > > a
> > >
> > > metadata request before the producer gets a chance. Of course, the
> > >
> > > way
> > >
> > > to
> > >
> > > avoid this, is to do as you suggested, and set
> > >
> > > the
> > >
> > > "allow_auto_topic_creation" field to false.
> > >
> > > I think the only thing we need to be careful with in this setup is
> > >
> > > without
> > >
> > > KIP 464, we can not use broker defaults for this topic. A user
> > >
> > > needs
> > >
> > > to
> > >
> > > specify the number of partition and replication factor in the
> > >
> > > config.
> > >
> > > An
> > >
> > > alternative to this is to have coded defaults for when these
> > >
> > > configs
> > >
> > > are
> > >
> > > unspecified, but it is not immediately apparent what these defaults
> > >
> > > should
> > >
> > > be.
> > >
> > >
> > > Thanks again for reading my KIP,
> > > Justine
> > >
> > >
> > > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> > >
> > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > >
> > > Thanks for the response!
> > > In my opinion, it would be better if the producer did not rely at
> > >
> > >
> > > all
> > >
> > >
> > > on the broker auto create feature as this is what we're aiming to
> > > deprecate. When requesting metadata we can set the
> > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > creation. Then if the topic is not existing, send a
> > >
> > >
> > > CreateTopicRequest.
> > >
> > > What do you think?
> > >
> > > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Currently the way it is implemented, the broker auto-creation
> > >
> > > configuration
> > >
> > > takes precedence. The producer will not use the CreateTopics
> > >
> > > request.
> > >
> > > (Technically it can--but the topic will already be created
> > >
> > > through
> > >
> > > the
> > >
> > > broker, so it will never try to create the topic.) It is possible
> > >
> > > to
> > >
> > > change this however, and I'd be happy to
> > >
> > > discuss
> > >
> > > the
> > >
> > > benefits of this alternative.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> > >
> > > mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP!
> > >
> > > In case auto creation is enabled on both the client and server,
> > >
> > > will
> > >
> > > the producer still use the AdminClient (CreateTopics request)
> > >
> > > to
> > >
> > > create topics? and not rely on the broker auto create. I'm
> > >
> > > guessing the
> > >
> > > answer is yes but can you make it explicit.
> > >
> > > Thank you
> > >
> > > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > >
> > > Hi,
> > > Just a friendly reminder to take a look at this KIP if you
> > >
> > >
> > > have
> > >
> > > the
> > >
> > > time.
> > >
> > > I was thinking about broker vs. client default precedence,
> > >
> > > and I
> > >
> > > think it
> > >
> > > makes sense to keep the broker as the default used when both
> > >
> > > client-side
> > >
> > > and broker-side defaults are configured. The idea is that
> > >
> > > there
> > >
> > > would be
> > >
> > > pretty clear documentation, and that many systems with
> > >
> > > configurations
> > >
> > > that
> > >
> > > the client could not change would likely have the auto-create
> > >
> > > default
> > >
> > > off.
> > >
> > > (In cloud for example).
> > >
> > >
> > > It also seems like in most cases, the consumer config
> > > ' allow. auto. create. topics ( http://allow.auto.create.topics/
> > >
> > >
> > > ) '
> > >
> > > was
> > >
> > > created to actually prevent
> > >
> > > the
> > >
> > > creation
> > >
> > > of
> > >
> > > topics, so the loss of creation functionality will not be a
> > >
> > > big
> > >
> > > problem.
> > >
> > > I'm happy to discuss any other compatibility problems or
> > >
> > > components
> > >
> > > of
> > >
> > > this KIP.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io )
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I was looking at this KIP again, and there is a decision I
> > >
> > > made
> > >
> > > that I
> > >
> > > think is worth discussing.
> > >
> > >
> > > In the case where both the broker and producer's
> > > 'auto.create.topics.enable' are set to true, we have to
> > >
> > >
> > > choose
> > >
> > > either
> > >
> > > the
> > >
> > > broker configs or the producer configs for the replication
> > > factor/partitions.
> > >
> > > Currently, the decision is to have the broker defaults take
> > >
> > > precedence.
> > >
> > > (It is easier to do this in the implementation.) It also
> > >
> > > makes
> > >
> > > some
> > >
> > > sense
> > >
> > > for this behavior to take precedence since this behavior
> > >
> > > already
> > >
> > > occurs as
> > >
> > > the default.
> > >
> > > However, I was wondering if it would be odd for those who
> > >
> > > can
> > >
> > > only
> > >
> > > see
> > >
> > > the
> > >
> > > producer side to set configs for replication
> > >
> > > factor/partitions
> > >
> > > and
> > >
> > > see
> > >
> > > different results. Currently the documentation for the
> > >
> > > config
> > >
> > > states
> > >
> > > that
> > >
> > > the config values are only used when the broker config is
> > >
> > > not
> > >
> > > enabled,
> > >
> > > but
> > >
> > > this might not always be clear to the user. Changing the
> > >
> > > code
> > >
> > > to
> > >
> > > have
> > >
> > > the
> > >
> > > producer's configurations take precedence is possible, but
> > >
> > > I
> > >
> > > was
> > >
> > > wondering
> > >
> > > what everyone thought.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Just a quick update--
> > >
> > > It seems that enabling both the broker and producer
> > >
> > > configs
> > >
> > > works
> > >
> > > fine,
> > >
> > > except that the broker configurations for partitions,
> > >
> > > replication
> > >
> > > factor
> > >
> > >
> > > take precedence.
> > > I don't know if that is something we would want to
> > >
> > >
> > > change, but
> > >
> > > I'll be
> > >
> > > updating the KIP for now to reflect this. Perhaps we would
> > >
> > > want to
> > >
> > > add more
> > >
> > > to the documentation of the the producer configs to
> > >
> > > clarify.
> > >
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for looking at the KIP. I can definitely add to
> > >
> > > the
> > >
> > > title
> > >
> > > to
> > >
> > > make
> > >
> > > it more clear.
> > >
> > > It makes sense that both configurations could be turned
> > >
> > > on
> > >
> > > since
> > >
> > > there
> > >
> > > are many cases where the user can not control the
> > >
> > > server-side
> > >
> > > configurations. I was a little concerned about how both
> > >
> > > interacting
> > >
> > > would
> > >
> > > work out -- if there would be to many requests for new
> > >
> > > topics,
> > >
> > > for
> > >
> > > example.
> > >
> > > But it since it does make sense to allow both
> > >
> > > configurations
> > >
> > > enabled, I
> > >
> > > will test out some scenarios and I'll change the KIP to
> > >
> > > support
> > >
> > > this.
> > >
> > > I also agree with documentation about distinguishing the
> > >
> > > differences. I
> > >
> > > was having some trouble with the wording but I like the
> > >
> > > phrases
> > >
> > > "server-side" and "client-side." That's a good
> > >
> > > distinction I
> > >
> > > can
> > >
> > > use
> > >
> > > when
> > >
> > > describing.
> > >
> > > I'll try to update the KIP soon keeping everyone's input
> > >
> > > in
> > >
> > > mind.
> > >
> > >
> > > Thanks,
> > > Justine
> > >
> > >
> > > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> > >
> > > cmccabe@ apache. org ( cmccabe@apache.org )
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP. This seems like a good step towards
> > >
> > > removing
> > >
> > > server-side topic auto-creation.
> > >
> > > We should add included "client-side" to the title of
> > >
> > > the KIP
> > >
> > > somewhere,
> > >
> > > to make it clear that we're talking about client-side
> > >
> > > auto
> > >
> > > creation.
> > >
> > > The KIP says:
> > >
> > > In order to automatically create topics with the
> > >
> > > producer, the
> > >
> > > producer's
> > >
> > > auto.create.topics.enable config must be set to true
> > >
> > > and
> > >
> > > the
> > >
> > > broker
> > >
> > > config should be set to false
> > >
> > > From a user's point of view, this seems
> > >
> > > counter-intuitive.
> > >
> > > In
> > >
> > > order to
> > >
> > > auto-create topics the broker's
> > >
> > > auto.create.topics.enable
> > >
> > > config
> > >
> > > should be
> > >
> > > set to false? It seems like the server-side
> > >
> > > auto-create is
> > >
> > > unrelated to
> > >
> > > the client-side auto-create. We could have both turned
> > >
> > > on
> > >
> > > (and
> > >
> > > I'm
> > >
> > > sure
> > >
> > > that in the real world, people will try this
> > >
> > > configuration...)
> > >
> > > There's no
> > >
> > > reason not to support this, I think.
> > >
> > > We should add some documentation explaining the
> > >
> > > difference
> > >
> > > between
> > >
> > > server-side and client-side auto-creation. Without
> > >
> > > documentation,
> > >
> > > an admin
> > >
> > > might think that they had disabled all forms of
> > >
> > > auto-creation by
> > >
> > > setting
> > >
> > > the -side setting to false-- but this is not the case,
> > >
> > > of
> > >
> > > course.
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > >
> > > Hi Dhruvil,
> > >
> > >
> > > Thanks for reading the KIP!
> > > That was the general idea for deprecation. We would
> > >
> > >
> > > log a
> > >
> > > warning
> > >
> > > when the
> > >
> > >
> > > config is enabled on the broker.
> > > I also believe that there would be a change to
> > >
> > >
> > > documentation.
> > >
> > > If there is anything else that should be done, please
> > >
> > > let
> > >
> > > me
> > >
> > > know!
> > >
> > > Justine
> > >
> > > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> > >
> > > dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP, this is great!
> > >
> > > Could you add some more information about what
> > >
> > > deprecating
> > >
> > > the
> > >
> > > broker
> > >
> > > configuration means? Would we log a warning in the
> > >
> > > logs
> > >
> > > when
> > >
> > > auto
> > >
> > > topic
> > >
> > > creation is enabled on the broker, for example?
> > >
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > >
> > > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I'd like to start a discussion thread for KIP-487. This KIP plans
> > >
> > > to
> > >
> > > deprecate the current system of
> > >
> > > auto-creating
> > >
> > > topics
> > >
> > > through requests to the metadata and give the
> > >
> > > producer the
> > >
> > > ability to
> > >
> > > automatically create topics instead.
> > >
> > > More information can be found here:
> > >
> > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > >
> > > https://cwiki.
> > >
> > >
> > > apache.org/confluence/display/KAFKA/ )
> > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > >
> > >
> > >
> > > Thank you,
> > > Justine Olshan
> > >
> > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by jiamei xie <ji...@gmail.com>.

Hi all
For https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer ,  It has not been updated for a long time. And I made some update, which has been pushed to https://github.com/apache/kafka/pull/8831

MetadataRequest has method Builder(List<String> topics, boolean allowAutoTopicCreation) by which we can set whether to enable allowAutoTopicCreation from producer.
By default, allowAutoTopicCreation on Producer is true. And only if when the allowAutoTopicCreation of Broker and Producer are true, the topic can be auto-created. 

Besides, the test cases are changed:
There are 4 cases for brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy, Check if the topic is created under these four cases.
     If brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy are true:  assertTrue(topicCreated)
     else : intercept[ExecutionException]

Looking forward to your feedback and comments. Thanks.

Best wishes
Jiamei Xie

On 2019/08/12 15:50:22, Harsha Chintalapani <ka...@harsha.io> wrote: 
> On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma <is...@juma.me.uk> wrote:
> 
> > Hi all,
> >
> > A few points:
> >
> > 1. I think the way backwards compatibility is being used here is not
> > correct. Any functionality that is only enabled if set via a config is
> > backwards compatible. People may disagree with the functionality or the
> > config, but it's not a backwards compatibility issue.
> >
> 
> We are talking about both broker and producer as a  single entity and run
> by the same team/users. Allowing newer producer to create topics on a older
> broker when auto.create.topics.enable set to false, breaks  server side
> contract that this config offered from the beginning.  IMO, it clearly
> isn't backward compatible. User who set auto.create.topic.enable on broker
> will not be the same who will turn it on producer side .
> 
> 
> > 2. It's an interesting question if auto topic creation via the producer
> > should be a server driven choice or not. I can see the desire to have a
> > server-driven default, but it seems like this is often application
> > specific. Because the functionality is trivially available via AdminClient
> > (released 2 years ago), it's not quite possible to control what
> > applications do without the use of ACLs or policies today.
> >
> >
> >
> Producers & consumers are the majority of the clients in Kafka ecosystem.
> Just because AdminClient shipped a while back that doesn't mean all users
> adopting to it. To this day lot more users are aware of Producer & Consumer
> APIs and running them in production compare to AdminClient.
> 
> 
> > 3. Changing the create topics request in this way is highly unintuitive in
> > my opinion and it relies on every client to pass the new field. For
> > example, if librdkafka added auto create functionality by relying on their
> > AdminClient, it would behave differently than what is proposed here.
> > Forcing every client to implement this change when calling auto create from
> > the producer specifically seems odd
> >
> 
> I am not sure why its unintuitive , protocols change. We add or upgrade the
> existing protocols all the time.
> 
> 
> Thanks,
> Harsha
> 
> .
> >
> > Ismael
> >
> > On Thu, Aug 8, 2019 at 11:51 AM Jun Rao <ju...@confluent.io> wrote:
> >
> > Hi, Justine,
> >
> > Thanks for the KIP. Overall, it seems to be a good improvement.
> >
> > However, I think Harsha's point seems reasonable. We had
> > auto.create.topics.enable config on the broker to allow admins to disable
> > topic creation from the producer/consumer clients before we had the
> > security feature. The need for that config is reduced with the security
> > feature, but may still be present since not all places have security
> > enabled. It's true that a non-secured environment is vulnerable to some
> > additional attacks, but producer/consumer are the most common way for a
> > user to interact with the broker. So, keeping that config for backward
> > compatibility could still be useful if it's not introducing too much effort
> > or extra confusion.
> >
> >
> > Here is a one potential alternative way that I was thinking. We add a new
> > field in the CreateTopicRequest to indicate whether it's from the producer
> > or not. If auto.create.topics.enable is false, CreateTopicRequest from the
> > producer will be rejected. We probably don't need to introduce the new
> > config (which seems a bit hard to explain) in the producer. Instead, the
> > new producer always uses MetadataRequest with AllowAutoTopicCreation set to
> > false to get the metadata and if the metadata is not present, send the new
> > CreateTopicRequest
> > (assuming the broker supports it) to try to create the topic
> > automatically. Whether the creation is allowed or not will be determined by
> > the broker. This will make the behavior backward compatible and we can
> > still achieve the main goal of the KIP, which is not relying on
> > MetadataRequest for topic creation. What do you think?
> >
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Aug 8, 2019 at 1:34 AM M. Manna <ma...@gmail.com> wrote:
> >
> > Hi,
> >
> >
> > If I may, perhaps you could simplify everything by using only
> > 'auto.create.topics.enable' as a value along with true. In other words,
> >
> >
> > the
> >
> > public interfaces section should only have
> >
> > [true,auto.create.topics.enable,
> >
> > false].
> >
> > The reason for this is that auto.create.topics.enable is already known to
> > users as a "Server-SIde" config. So all you are saying is
> >
> > a) To avoid day 1 impact, it will follow whatever
> >
> > auto.create.topics.enable
> >
> >
> > value is set.
> > b) False means - no client side topic creation
> > c) True means client side topic creation.
> >
> >
> > It saves creating 2 more new strings :). But not too expensive anyway.
> >
> > Also, when you deprecate auto.create.topics.enable - you must provide
> > sufficient logic to ensure that things like rolling upgrade doesn't
> > temporarily break anything. I apologise if you have already accounted for
> > this, but wanted to mention since I didn't notice this on the KIP.
> >
> > Let me know how this sounds.
> >
> > Regards,
> >
> > On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jo...@confluent.io>
> >
> > wrote:
> >
> > Hi Harsha,
> >
> > I think my message may have gotten lost in all the others.
> >
> > Two of the goals of this KIP are to 1) allow auto-creation on specific
> > clients when the broker default is false and 2) eventually replace the
> > broker config.
> >
> > In order to accomplish these two goals, we need the producer to be able
> >
> > to
> >
> >
> > create topics despite the broker config. (How can we replace a function
> > when we rely on it?)
> > I think at this point we have a fundamental disagreement in what we
> >
> >
> > should
> >
> >
> > allow the producer to do.
> > In my previous message I mentioned a config that would allow for the
> >
> >
> > broker
> >
> > to prevent producer auto-creation. (It would be disabled by default.)
> >
> > It
> >
> > would fix your issue for now, but could lead to more complications
> >
> > later.
> >
> >
> > Thank you,
> > Justine
> >
> >
> > On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz>
> >
> > wrote:
> >
> > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> >
> > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe <
> >
> > cmccabe@apache.org
> >
> > wrote:
> >
> > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> >
> >
> > Hi Colin,
> > "Hmm... I'm not sure I follow. Users don't have to build their own
> > tooling, right? They can use any of the shell scripts that we've
> >
> >
> > shipped
> >
> > in
> >
> > the last few releases. For example, if any of your users run it,
> >
> > this
> >
> > shell
> >
> > script will delete all of the topics from your non-security-enabled
> > cluster:
> >
> >
> > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > --bootstrap-server localhost:9092 --list 2>/dev/null
> > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > --bootstrap-server localhost:9092 --delete
> > --topic
> >
> >
> > They will need to fill in the correct bootstrap servers list, of
> >
> > course,
> >
> > not localhost. This deletion script will work on some pretty old
> >
> > brokers,
> >
> > even back to the 0.10 releases. It seems a little odd to trust your
> >
> > users
> >
> > with this power, but not trust them to avoid changing a particular
> > configuration key."
> >
> > The above will blocked by the server if we set delete.topic.enable
> >
> > to
> >
> > false and thats exactly what I am asking for.
> >
> > Hi Harsha,
> >
> > I was wondering if someone was going to bring up that configuration
> >
> > :)
> >
> > it's an interesting complication, but globally disabling topic
> >
> > deletion
> >
> > is
> >
> > not very practical for most use-cases.
> >
> > In any case, there are plenty of other bad things that users with
> >
> > full
> >
> > permissions can do that aren't blocked by any server configuration.
> >
> > For
> >
> > example, they can delete every record in every topic. I can write a
> >
> > script
> >
> > for that too, and there's no server configuration you can set to
> >
> > disable
> >
> > it. Or I could simply create hundreds of thousands of topics, until
> >
> > cluster
> >
> > performance becomes unacceptable (this will be even more of a
> >
> > problem
> >
> > if
> >
> > someone configured delete.topic.enable as false). Or publish bad
> >
> > data
> >
> > to
> >
> > every topic, etc. etc.
> >
> > The point I'm trying to make here is that you can't rely on these
> >
> > kind
> >
> > of
> >
> > server-side configurations for security. At most, they're a way to
> >
> > set
> >
> > up
> >
> > certain very simple policies. But the policies are so simple that
> >
> > they're
> >
> > hardly ever useful any more.
> >
> > For example, if the problem you want to solve is that you want a
> >
> > user
> >
> > to
> >
> > only be able to create 50 topics and not delete anyone else's
> >
> > topics,
> >
> > you
> >
> > can solve that with a CreateTopicsPolicy that limits the number of
> >
> > topics,
> >
> > and some ACLs. There's no combination of auto.create.topics.enable
> >
> > and
> >
> > delete.topic.enable that will help here.
> >
> > Hi Colin,
> >
> > Well you gave the example that a user can delete the
> >
> > topics
> >
> > just by running that script :).
> >
> > I understand there are open APIs in Kafka and can lead to rogue
> >
> > clients
> >
> > taking advantage of it without proper security in place.
> >
> > What I am asking so far in this thread is , this KIP is changing
> >
> > the
> >
> > producer behavior and its not backward compatible.
> >
> > The change is backwards compatible. The default will still be
> >
> > server-side
> >
> > topic auto-creation, just like now.
> >
> > You will have to specifically change the producer config to get the
> >
> > new
> >
> > behavior.
> >
> > I disagree. Today server can turn off the topic creation neither
> >
> > producer
> >
> > or consumer can create a topic. With this KIP , producer can create a
> >
> > topic
> >
> > by turning on client side config when server side config is turned
> >
> > off.
> >
> > We can still achieve
> >
> > the main goal of the KIP which is to change MetadataRequest
> >
> > creating
> >
> > topics and send a CreateTopicRequest from Producer and also keep
> >
> > the
> >
> > server
> >
> > side config to have precedence. This KIP originally written to
> >
> > have
> >
> > server
> >
> > side preference and there is not much explanation why it changed to
> >
> > have
> >
> > Producer side preference.
> >
> > Arguing that AdminClient can do that and so we are going to make
> >
> > Producer
> >
> > do the same doesn't make sense.
> >
> > "The downside is that if we wanted to check a server side
> >
> > configuration
> >
> > before sending the create topics request, the code would be more
> >
> > complex.
> >
> > The behavior would also not be consistent with how topic
> >
> > auto-creation
> >
> > is
> >
> > handled in Kafka Streams."
> >
> > I am not sure why you need to check server side configuration
> >
> > before
> >
> > sending create topics request. A user enables producer side config
> >
> > to
> >
> >
> > create topics.
> > Producer sends a request to the broker and if the broker has
> > auto.topic.create.enable to true (default) it will allow creation
> >
> >
> > of
> >
> > topics. If it set to false it returns error back to the client.
> >
> > auto.topic.create.enable has never affected CreateTopicsRequest. If
> >
> > you
> >
> > submit a CreateTopicsRequest and you are authorized, a topic will
> >
> > be
> >
> > created, regardless of what the value of auto.topic.create.enable
> >
> > is.
> >
> > This
> >
> > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I
> >
> > think?)
> >
> > I don't see how this behavior will be different in Kafka streams.
> >
> > By
> >
> > default server allows the topic creation and with this KIP, It will
> >
> > only
> >
> > allow creation of topic when both producer and server side are
> >
> > turned
> >
> > on.
> >
> > Its exactly the same behavior in KIP-361.
> >
> > I suggest running a streams job as a test. It creates the topics it
> >
> > needs
> >
> > using AdminClient. The value of auto.topic.create.enable is not
> >
> > relevant.
> >
> > Whether it is true or false, the topics will still be created.
> >
> > "In general, it would be nice if we could keep the security and
> >
> > access
> >
> > control model simple and not introduce a lot of special cases and
> > exceptions. Kafka has basically converged on using ACLs and
> > CreateTopicPolicy classes to control who can create topics and
> >
> > where.
> >
> > Adding more knobs seems like a step backwards, especially when the
> >
> > proposed
> >
> > knobs don't work consistently across components, and don't provide
> >
> > true
> >
> > security." This is not about access control at all. Shipping sane
> >
> > defaults
> >
> > should be prioritized.
> >
> > I don't think the default is really the question here. I think
> >
> > everyone
> >
> > agrees that the default for client-side auto-topic creation should
> >
> > be
> >
> > off.
> >
> > My point goes back to KIP-361 why we kept the server side config to
> >
> > have
> >
> > preference. We can keep bringing back the discussion of security
> >
> > policies
> >
> > but a producer client overiding server side setting is not just
> >
> > security
> >
> > policy issue .
> >
> > "Producer client can override server side setting and not consumer
> >
> > and
> >
> > also producer can only override creation of topic but no the
> >
> > replication
> >
> > and partitions. " This doesn't make sense to me and why we want to
> > introduce such a confusing behavior.
> >
> > The scenarios that you're describing (such as dealing with a poorly
> > configured client that tries to create topics it should not) do
> >
> > seem
> >
> > to
> >
> > be
> >
> > about access control.
> >
> > We keep talking about CreateTopicPolicy and yet we don't have
> >
> > default
> >
> > one
> >
> > and asking every user of Kafka implement their own doesn't make
> >
> > sense
> >
> > here.
> >
> > The point of CreateTopicPolicy is exactly that you can implement
> >
> > your
> >
> > own,
> >
> > though. It's a way of customizing what the policy is in your
> >
> > specific
> >
> > cluster.
> >
> > I agree that most people don't want to write a custom policy. But
> >
> > most
> >
> > of
> >
> > those people are satisfied with just setting up ACLs to set up
> >
> > simple
> >
> > policies like this user can create topics, that user can't, etc.
> >
> > That's
> >
> > why
> >
> > I suggested adding support for ACLs to insecure clusters might be
> >
> > helpful.
> >
> > Also, just as a side note, wouldn't it would be impossible for us
> >
> > to
> >
> > specify a default CreateTopicsPolicy that did anything at this
> >
> > point
> >
> > anyway
> >
> > without breaking backwards compatibility? Maybe I'm
> >
> > misunderstanding
> >
> > the
> >
> > proposal.
> >
> > I am asking about exact behavior that we shipped for consumer side
> >
> > https:/
> >
> > / cwiki. apache. org/ confluence/ display/ KAFKA/
> >
> > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> >
> >
> > (
> > https://cwiki.apache.org/confluence/display/KAFKA/
> >
> >
> > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> >
> > )
> >
> > I still didn't get any response why this behavior shouldn't be
> >
> > exactly
> >
> > like Kafka consumer and why do we want to have producer to
> >
> > overrider
> >
> > server
> >
> > side config and while not allowing consumer to do so. We are not
> >
> > even
> >
> > allowing the same contract and this will cause more confusion from
> >
> > the
> >
> > users standpoint.
> >
> > Hmm. The consumer already has a way to override the server side
> > configuration. See KIP-361: Add Consumer Configuration to Disable
> >
> > Auto
> >
> > Topic Creation. This lets the consumer skip auto-creating topics,
> >
> > even
> >
> > if
> >
> > auto-creation is enabled on the broker.
> >
> > Yes that's what I am saying and it doesn't allow topic creation if
> >
> > the
> >
> > server side auto-creation is disabled and in consumer its enabled.
> >
> > I
> >
> > would like to see the exact behavior for producer as stated in
> >
> > KIP-361.
> >
> > To be fair, the KIP should probably discuss why we don't want to
> >
> > implement
> >
> > client-side topic creation in the consumer in "rejected
> >
> > alternatives."
> >
> > Maybe Justine can add more context here in the KIP.
> >
> > The last time we talked about this, the reasoning was that we
> >
> > wanted
> >
> > to
> >
> > eventually get rid of consumer-side auto-topic creation entirely,
> >
> > and
> >
> > so
> >
> > it
> >
> > wasn't worth implementing an improved way of doing it. Producer
> >
> > side
> >
> > auto
> >
> > topic-creation, in contrast, will probably stick around, although
> >
> > we'd
> >
> > like
> >
> > to deprecate and remove the problematic server-side mechanism over
> >
> > time.
> >
> > We should implement the producer side topic creation with proper
> >
> > create
> >
> > topic request and I am in favor of this KIP.
> >
> > All I am asking is don't make the producer to override server side
> >
> > config
> >
> > and keep it similar to KIP-361 just like consumer, it honors what
> >
> > set
> >
> > on
> >
> > server side which takes the precedence. Changing this behavior in producer
> > is backward incompatible and will catch users by surprise.
> >
> > All arguments for enforcing producer side overriding config can
> >
> > still
> >
> > be
> >
> > achieved. Server side auto creation turned on by default and any
> >
> > new
> >
> > producer client setting their auto creation config to true will
> >
> > create
> >
> > topics in default behavior and any user who set server side to
> >
> > false
> >
> > and
> >
> > upgrades to latest kafka with this KIP will not see any unwanted
> >
> > behavior
> >
> > from clients. IMO this is not a unreasonable request on this KIP
> >
> > and
> >
> > this
> >
> > requested behavior is exactly what the KIP initially proposed.
> >
> > Thanks,
> >
> > Harsha
> >
> >
> > best,
> > Colin
> >
> >
> > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache.
> >
> > org (
> >
> > cmccabe@apache.org ) > wrote:
> >
> > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> >
> > Not sure how the AdminClient applies here, It is an external API
> >
> > and
> >
> > is
> >
> > not part of KafkaProducer so any user who updates to latest version
> >
> > of
> >
> >
> > Kafka with this feature get to create the topics.
> > They have to build a tooling around AdminClient allow themselves to
> >
> >
> > create
> >
> > topics.
> >
> > Hi Harsha,
> >
> > Hmm... I'm not sure I follow. Users don't have to build their own
> >
> > tooling,
> >
> > right? They can use any of the shell scripts that we've shipped in
> >
> > the
> >
> > last
> >
> > few releases. For example, if any of your users run it, this shell
> >
> > script
> >
> > will delete all of the topics from your non-security-enabled
> >
> > cluster:
> >
> >
> > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > --bootstrap-server localhost:9092 --list 2>/dev/null
> > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > --bootstrap-server localhost:9092 --delete
> > --topic
> >
> >
> > They will need to fill in the correct bootstrap servers list, of
> >
> > course,
> >
> > not localhost. This deletion script will work on some pretty old
> >
> > brokers,
> >
> > even back to the 0.10 releases. It seems a little odd to trust your
> >
> > users
> >
> > with this power, but not trust them to avoid changing a particular
> > configuration key.
> >
> > There is no behavior in Kafka producer that allowed users to delete
> >
> > the
> >
> > topics or delete the records. So citing them as an example doesn't
> >
> > makes
> >
> > sense in this context.
> >
> > I think Kafka Streams is relevant here. After all, it's software
> >
> > that
> >
> > we
> >
> > ship as part of the official Kafka release. And it auto-creates
> >
> > topics
> >
> > even
> >
> > when auto.create.topics.enable is set to false on the broker.
> >
> > I think that auto.create.topics.enable was never intended as a
> >
> > security
> >
> > setting to control access. It was intended as a way to disable the
> > broker-side auto-create feature specifically, because there were
> >
> > some
> >
> > problems with that specific feature. Broker-side auto-creation is
> > frustrating because it's on by default, and because it can
> >
> > auto-create
> >
> > topics even if the producer or consumer didn't explicitly ask for
> >
> > that.
> >
> > Neither of those problems applies to this KIP: producers have to
> > specifically opt in, and it won't be on by default. Basically, we
> >
> > think
> >
> > that client-side auto-creation is actually a lot better-- hence
> >
> > this
> >
> > KIP
> >
> > :)
> >
> >
> > But there is
> > a functionality which allowed creation of topics if they don't
> >
> >
> > exist
> >
> > in
> >
> > the
> >
> > cluster and this behavior could be controlled by a config on the
> >
> > server
> >
> > side. Now with this KIP we are allowing producer to make that
> >
> > decision
> >
> > without any gateway on the server via configs. Any user who is not
> >
> > aware
> >
> > of
> >
> >
> > such changes
> > can accidentally create these topics and we are essentially
> >
> >
> > removing
> >
> > a
> >
> > config that exists in brokers today to block this accidental
> >
> > creation
> >
> > and
> >
> > allowing clients to take control.
> >
> > Again, I hope I'm not misinterpreting, but I don't see how this can
> >
> > be
> >
> > turned on accidentally. The user would have to specifically turn
> >
> > this
> >
> > on
> >
> > in
> >
> > the producer by setting the configuration key.
> >
> > I still didn't get any positives of not having server side configs?
> >
> > if
> >
> > you
> >
> > want to turn them on and allow any client to create topics set the
> >
> > default
> >
> >
> > to true
> > and allow users who doesn't want to have this feature let them turn
> >
> >
> > them
> >
> > off. It will be the exact behavior as it is today , as far as
> >
> > producer
> >
> > is
> >
> >
> > concerned. I am not
> > understanding why not having server side configs to gateways such a
> >
> >
> > hard
> >
> > requirement and this behavior exists today. As far I am concerned
> >
> > this
> >
> > breaks the backward compatibility.
> >
> > The downside is that if we wanted to check a server side
> >
> > configuration
> >
> > before sending the create topics request, the code would be more
> >
> > complex.
> >
> > The behavior would also not be consistent with how topic
> >
> > auto-creation
> >
> > is
> >
> > handled in Kafka Streams.
> >
> > In general, it would be nice if we could keep the security and
> >
> > access
> >
> > control model simple and not introduce a lot of special cases and
> > exceptions. Kafka has basically converged on using ACLs and
> > CreateTopicPolicy classes to control who can create topics and
> >
> > where.
> >
> > Adding more knobs seems like a step backwards, especially when the
> >
> > proposed
> >
> > knobs don't work consistently across components, and don't provide
> >
> > true
> >
> > security.
> >
> > Maybe we should support an insecure mode where there are still
> >
> > users
> >
> > and
> >
> > ACLs. That would help people who don't want to set up Kerberos,
> >
> > etc.
> >
> > but
> >
> > still want to protect against misconfigured clients or accidents.
> >
> > Hadoop
> >
> > has something like this, and I think it was useful. NFS also
> >
> > supported
> >
> > (supports?) a mode where you just pass whatever user ID you want
> >
> > and
> >
> > the
> >
> > system believes you. These things clearly don't protect against
> >
> > malicious
> >
> > users, but they can help set up policies when needed.
> >
> >
> > best,
> > Colin
> >
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache.
> >
> > org (
> >
> > cmccabe@apache.org ) > wrote:
> >
> > Hi Harsha,
> >
> > Thanks for taking a look.
> >
> > I'm not sure I follow the discussion about AdminClient.
> >
> > KafkaAdminClient
> >
> > has been around for about 2 years at this point as a public class.
> >
> > There
> >
> > are many programs that use it to automatically create topics. Kafka
> > Streams does this, for example. If any of your users run Kafka
> >
> > Streams,
> >
> > they will be auto-creating topics, regardless of what setting you
> >
> > use
> >
> > for
> >
> > auto.create.topics.enable.
> >
> > A big part of the annoyance of auto-topic creation right now is
> >
> > that
> >
> > it is
> >
> > on by default. The new configuration proposed by KIP-487 wouldn't
> >
> > be.
> >
> > Users would have to explicitly opt in to the new behavior of
> >
> > client-side
> >
> > topic creation. If you run without security, you're already
> >
> > putting a
> >
> > huge
> >
> > amount of trust in your users. For example, you trust them not to
> >
> > delete
> >
> > records with the kafka-delete-records. sh (
> >
> > http://kafka-delete-records.
> >
> >
> > sh/
> > ) command, or delete topics with kafka-topics. sh (
> >
> >
> > http://kafka-topics.
> >
> >
> > sh/
> > ). Trusting them not to set a certain config value seems minor in
> > comparison, right?
> >
> >
> >
> > best,
> > Colin
> >
> >
> > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> >
> >
> > Hi,
> > Even with policies one needs to implement that, so for every
> >
> >
> > user who
> >
> > doesn't want a producer to create topics or have limits around
> >
> > partitions &
> >
> > replication factor they have to implement a policy. The KIP is
> >
> > changing
> >
> > the behavior , it might not be introducing
> >
> > the
> >
> > new functionality but it will enable producers to override the
> >
> > create
> >
> > topic
> >
> > config settings on the broker. What I am asking for to provide a
> >
> > config
> >
> > that will disable auto creation of topics and if its enabled set
> >
> > some
> >
> > sane
> >
> > defaults so that clients can create a topic with in those limits. I
> >
> > don't
> >
> >
> > see how this not related to this KIP.
> > If the server config options as I suggested doesn't interest
> >
> >
> > you at
> >
> >
> > least have a default CreateTopicPolices in place.
> > To give an example, In our environment we disable the
> > auto.create.topic.enable and force users to go through a
> >
> >
> > centralized
> >
> > service as we want capture more details about the topic creation
> >
> > and
> >
> > requirements. With this KIP, a producer can create a topic with no
> >
> > bounds.
> >
> > Another example max.message.size we define that at cluster level
> >
> > and one
> >
> > can override max.messsage.size at topic level, any misconfiguration
> >
> > at
> >
> > this
> >
> > will cause service degradation. Its not always about the rogue
> >
> > clients,
> >
> > users can easily misconfigure and can cause an outage. Again we can
> >
> > talk
> >
> > about CreateTopicPolicy but without having a
> >
> > default
> >
> > implementation and asking everyone to implement their own while
> >
> > changing
> >
> > the behavior in producer doesn't make sense to me.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me.
> >
> > uk (
> >
> > ismael@juma.me.uk ) >
> >
> > wrote:
> >
> > Hi Harsha,
> >
> > I mentioned policies and the authorizer. For example, with
> > CreateTopicPolicy, you can implement the limits you describe. If
> >
> > you
> >
> > have
> >
> > ideas of how that should be improved, please submit a KIP. My
> >
> > point is
> >
> > that
> >
> > this KIP is not introducing any new functionality with regards to
> >
> > what
> >
> > rogue clients can do. It's using the existing protocol that is
> >
> > already
> >
> > exposed via the AdminClient. So, I don't think we need to address
> >
> > it in
> >
> > this KIP. Does that make sense?
> >
> > Ismael
> >
> > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> >
> > kafka@ harsha. io ( kafka@harsha.io ) >
> >
> > wrote:
> >
> >
> > Ismael,
> > Sure AdminClient can do that and we should've shipped a config or
> >
> >
> > use
> >
> > the
> >
> > existing one to block that. Not all users are yet to upgrade to
> >
> > AdminClient
> >
> > and start using that to cause issues yet. In shared environment we
> >
> > should
> >
> > allow server to set sane defaults and not allow every client to go
> >
> > ahead
> >
> > create random no.of topic/partitions and replication factor. Even
> >
> > if
> >
> > the
> >
> > users want to allow topic creation proposed in the KIP , it makes
> >
> > sense to
> >
> > have some guards against the no.of partitions and replication
> >
> > factor.
> >
> > Authorizer is not always an answer to block requests and having
> >
> > users
> >
> > set
> >
> > server side configs to protect a multi-tenant environment is
> >
> > required.
> >
> > In a
> >
> > non-secure environment Authorizer is a blunt instrument either you
> >
> > end
> >
> > up
> >
> >
> > blocking everyone or allowing everyone.
> > I am asking to have server side that allow clients to create
> >
> >
> > topics or
> >
> > not
> >
> > , if they are allowed set a ceiling on max no.of partitions and
> > replication-factor.
> >
> > -Harsha
> >
> > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com (
> >
> > ismaelj@gmail.com
> >
> > )
> >
> > wrote:
> >
> > Harsha,
> >
> > Rogue clients can use the admin client to create topics and
> >
> > partitions.
> >
> > ACLs and policies can help in that case as well as this one.
> >
> > Ismael
> >
> > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha.
> >
> > io
> >
> > (
> >
> > kafka@harsha.io ) >
> >
> > wrote:
> >
> >
> > Hi Justine,
> > Thanks for the KIP.
> > "When server-side auto-creation is disabled, client-side
> >
> >
> > auto-creation
> >
> >
> > will try to use client-side configurations"
> > If I understand correctly, this KIP is removing any server-side
> >
> >
> > blocking
> >
> >
> > client auto creation of topic?
> > if so this will present potential issue of rogue client creating
> >
> >
> > ton of
> >
> > topic-partitions and potentially bringing down the service for
> >
> > everyone
> >
> > or
> >
> >
> > degrade the service itself.
> > By reading the KIP its not clear to me that there is a clear way to
> >
> >
> > block
> >
> > auto creation topics of all together from clients by server side
> >
> > config.
> >
> > Server side configs of default topic, partitions should take higher
> > precedence and client shouldn't be able to create a topic with
> >
> > higher
> >
> > no.of
> >
> > partitions, replication than what server config specifies.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> >
> > Hi all,
> > I made some changes to the KIP. Hopefully this configuration change
> >
> >
> > will
> >
> > make things a little clearer.
> >
> > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> >
> > https://cwiki.
> >
> >
> > apache.org/confluence/display/KAFKA/ )
> > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> >
> >
> > Please let me know if you have any feedback or questions!
> >
> >
> > Thank you,
> > Justine
> >
> >
> > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache.
> >
> > org (
> >
> > cmccabe@apache.org ) >
> >
> > wrote:
> >
> > Hi Mickael,
> >
> > I think you bring up a good point. It would be better if we didn't
> >
> > ever
> >
> > have to set up client-side configuration for this feature, and
> >
> > KIP-464
> >
> > would let us skip this entirely.
> >
> > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> >
> >
> > Hi Mickael,
> > I agree that KIP-464 works on newer brokers, but I was a bit
> >
> >
> > worried
> >
> > how
> >
> > things would play out on older brokers that* do not *have KIP 464
> >
> > included.
> >
> > Is it enough to throw an error in this case when producer configs
> >
> > are
> >
> > not
> >
> > specified?
> >
> > I think the right thing to do would be to log an error message in
> >
> > the
> >
> > client. We will need to have that capability in any case, to cover
> > scenarios like the client trying to auto-create a topic that they
> >
> > don't
> >
> > have permission to create. Or a client trying to create a topic on
> >
> > a
> >
> > broker
> >
> > so old that CreateTopicsRequest is not supported.
> >
> > The big downside to relying on KIP-464 is that it is a very recent
> >
> > feature
> >
> > -- so recent that it hasn't even made its way to any official
> >
> > Apache
> >
> > release. It's scheduled for the upcoming 2.4 release in a few
> >
> > months.
> >
> > So if you view this KIP as a step towards removing broker-side
> > auto-create, you might want to support older brokers just to
> >
> > accelerate
> >
> > adoption, and hasten the day when we can finally flip broker-side
> > auto-create to off (or even remove it entirely).
> >
> > I have to agree, though, that having client-side configurations for
> >
> > number
> >
> > of partitions and replication factor is messy. Maybe it would be
> >
> > worth
> >
> > it
> >
> > to restrict support to post-KIP-464 brokers, if we could avoid
> >
> > creating
> >
> > more configs.
> >
> >
> > best,
> > Colin
> >
> >
> > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> >
> > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> >
> > wrote:
> >
> > Hi Justine,
> >
> > We can rely on KIP-464 which allows to omit the partition count or
> > replication factor when creating a topic. In that case, the broker
> >
> > defaults
> >
> > are used.
> >
> > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> >
> > Michael,
> > That makes sense to me!
> > To clarify, in the current state of the KIP, the producer does not
> >
> >
> > rely
> >
> > on
> >
> > the broker to autocreate--if the broker's config is disabled, then
> >
> > the
> >
> > producer can autocreate on its own with a create topic request (the
> >
> > same
> >
> >
> > type of request the admin client uses).
> > However, if both configs are enabled, the broker will autocreate
> >
> >
> > through
> >
> > a
> >
> > metadata request before the producer gets a chance. Of course, the
> >
> > way
> >
> > to
> >
> > avoid this, is to do as you suggested, and set
> >
> > the
> >
> > "allow_auto_topic_creation" field to false.
> >
> > I think the only thing we need to be careful with in this setup is
> >
> > without
> >
> > KIP 464, we can not use broker defaults for this topic. A user
> >
> > needs
> >
> > to
> >
> > specify the number of partition and replication factor in the
> >
> > config.
> >
> > An
> >
> > alternative to this is to have coded defaults for when these
> >
> > configs
> >
> > are
> >
> > unspecified, but it is not immediately apparent what these defaults
> >
> > should
> >
> > be.
> >
> >
> > Thanks again for reading my KIP,
> > Justine
> >
> >
> > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> >
> > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> >
> > wrote:
> >
> > Hi Justine,
> >
> >
> > Thanks for the response!
> > In my opinion, it would be better if the producer did not rely at
> >
> >
> > all
> >
> >
> > on the broker auto create feature as this is what we're aiming to
> > deprecate. When requesting metadata we can set the
> > "allow_auto_topic_creation" field to false to avoid the broker auto
> > creation. Then if the topic is not existing, send a
> >
> >
> > CreateTopicRequest.
> >
> > What do you think?
> >
> > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Currently the way it is implemented, the broker auto-creation
> >
> > configuration
> >
> > takes precedence. The producer will not use the CreateTopics
> >
> > request.
> >
> > (Technically it can--but the topic will already be created
> >
> > through
> >
> > the
> >
> > broker, so it will never try to create the topic.) It is possible
> >
> > to
> >
> > change this however, and I'd be happy to
> >
> > discuss
> >
> > the
> >
> > benefits of this alternative.
> >
> >
> > Thank you,
> > Justine
> >
> >
> > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> >
> > mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP!
> >
> > In case auto creation is enabled on both the client and server,
> >
> > will
> >
> > the producer still use the AdminClient (CreateTopics request)
> >
> > to
> >
> > create topics? and not rely on the broker auto create. I'm
> >
> > guessing the
> >
> > answer is yes but can you make it explicit.
> >
> > Thank you
> >
> > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> >
> > Hi,
> > Just a friendly reminder to take a look at this KIP if you
> >
> >
> > have
> >
> > the
> >
> > time.
> >
> > I was thinking about broker vs. client default precedence,
> >
> > and I
> >
> > think it
> >
> > makes sense to keep the broker as the default used when both
> >
> > client-side
> >
> > and broker-side defaults are configured. The idea is that
> >
> > there
> >
> > would be
> >
> > pretty clear documentation, and that many systems with
> >
> > configurations
> >
> > that
> >
> > the client could not change would likely have the auto-create
> >
> > default
> >
> > off.
> >
> > (In cloud for example).
> >
> >
> > It also seems like in most cases, the consumer config
> > ' allow. auto. create. topics ( http://allow.auto.create.topics/
> >
> >
> > ) '
> >
> > was
> >
> > created to actually prevent
> >
> > the
> >
> > creation
> >
> > of
> >
> > topics, so the loss of creation functionality will not be a
> >
> > big
> >
> > problem.
> >
> > I'm happy to discuss any other compatibility problems or
> >
> > components
> >
> > of
> >
> > this KIP.
> >
> >
> > Thank you,
> > Justine
> >
> >
> > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io )
> >
> > wrote:
> >
> > Hello all,
> >
> > I was looking at this KIP again, and there is a decision I
> >
> > made
> >
> > that I
> >
> > think is worth discussing.
> >
> >
> > In the case where both the broker and producer's
> > 'auto.create.topics.enable' are set to true, we have to
> >
> >
> > choose
> >
> > either
> >
> > the
> >
> > broker configs or the producer configs for the replication
> > factor/partitions.
> >
> > Currently, the decision is to have the broker defaults take
> >
> > precedence.
> >
> > (It is easier to do this in the implementation.) It also
> >
> > makes
> >
> > some
> >
> > sense
> >
> > for this behavior to take precedence since this behavior
> >
> > already
> >
> > occurs as
> >
> > the default.
> >
> > However, I was wondering if it would be odd for those who
> >
> > can
> >
> > only
> >
> > see
> >
> > the
> >
> > producer side to set configs for replication
> >
> > factor/partitions
> >
> > and
> >
> > see
> >
> > different results. Currently the documentation for the
> >
> > config
> >
> > states
> >
> > that
> >
> > the config values are only used when the broker config is
> >
> > not
> >
> > enabled,
> >
> > but
> >
> > this might not always be clear to the user. Changing the
> >
> > code
> >
> > to
> >
> > have
> >
> > the
> >
> > producer's configurations take precedence is possible, but
> >
> > I
> >
> > was
> >
> > wondering
> >
> > what everyone thought.
> >
> >
> > Thank you,
> > Justine
> >
> >
> > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Just a quick update--
> >
> > It seems that enabling both the broker and producer
> >
> > configs
> >
> > works
> >
> > fine,
> >
> > except that the broker configurations for partitions,
> >
> > replication
> >
> > factor
> >
> >
> > take precedence.
> > I don't know if that is something we would want to
> >
> >
> > change, but
> >
> > I'll be
> >
> > updating the KIP for now to reflect this. Perhaps we would
> >
> > want to
> >
> > add more
> >
> > to the documentation of the the producer configs to
> >
> > clarify.
> >
> >
> > Thank you,
> > Justine
> >
> >
> > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Hi Colin,
> >
> > Thanks for looking at the KIP. I can definitely add to
> >
> > the
> >
> > title
> >
> > to
> >
> > make
> >
> > it more clear.
> >
> > It makes sense that both configurations could be turned
> >
> > on
> >
> > since
> >
> > there
> >
> > are many cases where the user can not control the
> >
> > server-side
> >
> > configurations. I was a little concerned about how both
> >
> > interacting
> >
> > would
> >
> > work out -- if there would be to many requests for new
> >
> > topics,
> >
> > for
> >
> > example.
> >
> > But it since it does make sense to allow both
> >
> > configurations
> >
> > enabled, I
> >
> > will test out some scenarios and I'll change the KIP to
> >
> > support
> >
> > this.
> >
> > I also agree with documentation about distinguishing the
> >
> > differences. I
> >
> > was having some trouble with the wording but I like the
> >
> > phrases
> >
> > "server-side" and "client-side." That's a good
> >
> > distinction I
> >
> > can
> >
> > use
> >
> > when
> >
> > describing.
> >
> > I'll try to update the KIP soon keeping everyone's input
> >
> > in
> >
> > mind.
> >
> >
> > Thanks,
> > Justine
> >
> >
> > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> >
> > cmccabe@ apache. org ( cmccabe@apache.org )
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP. This seems like a good step towards
> >
> > removing
> >
> > server-side topic auto-creation.
> >
> > We should add included "client-side" to the title of
> >
> > the KIP
> >
> > somewhere,
> >
> > to make it clear that we're talking about client-side
> >
> > auto
> >
> > creation.
> >
> > The KIP says:
> >
> > In order to automatically create topics with the
> >
> > producer, the
> >
> > producer's
> >
> > auto.create.topics.enable config must be set to true
> >
> > and
> >
> > the
> >
> > broker
> >
> > config should be set to false
> >
> > From a user's point of view, this seems
> >
> > counter-intuitive.
> >
> > In
> >
> > order to
> >
> > auto-create topics the broker's
> >
> > auto.create.topics.enable
> >
> > config
> >
> > should be
> >
> > set to false? It seems like the server-side
> >
> > auto-create is
> >
> > unrelated to
> >
> > the client-side auto-create. We could have both turned
> >
> > on
> >
> > (and
> >
> > I'm
> >
> > sure
> >
> > that in the real world, people will try this
> >
> > configuration...)
> >
> > There's no
> >
> > reason not to support this, I think.
> >
> > We should add some documentation explaining the
> >
> > difference
> >
> > between
> >
> > server-side and client-side auto-creation. Without
> >
> > documentation,
> >
> > an admin
> >
> > might think that they had disabled all forms of
> >
> > auto-creation by
> >
> > setting
> >
> > the -side setting to false-- but this is not the case,
> >
> > of
> >
> > course.
> >
> >
> > best,
> > Colin
> >
> >
> > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> >
> > Hi Dhruvil,
> >
> >
> > Thanks for reading the KIP!
> > That was the general idea for deprecation. We would
> >
> >
> > log a
> >
> > warning
> >
> > when the
> >
> >
> > config is enabled on the broker.
> > I also believe that there would be a change to
> >
> >
> > documentation.
> >
> > If there is anything else that should be done, please
> >
> > let
> >
> > me
> >
> > know!
> >
> > Justine
> >
> > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> >
> > dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP, this is great!
> >
> > Could you add some more information about what
> >
> > deprecating
> >
> > the
> >
> > broker
> >
> > configuration means? Would we log a warning in the
> >
> > logs
> >
> > when
> >
> > auto
> >
> > topic
> >
> > creation is enabled on the broker, for example?
> >
> >
> > Thanks,
> > Dhruvil
> >
> >
> > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Hello all,
> >
> > I'd like to start a discussion thread for KIP-487. This KIP plans
> >
> > to
> >
> > deprecate the current system of
> >
> > auto-creating
> >
> > topics
> >
> > through requests to the metadata and give the
> >
> > producer the
> >
> > ability to
> >
> > automatically create topics instead.
> >
> > More information can be found here:
> >
> > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> >
> > https://cwiki.
> >
> >
> > apache.org/confluence/display/KAFKA/ )
> > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> >
> >
> >
> > Thank you,
> > Justine Olshan
> >
> >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Harsha Chintalapani <ka...@harsha.io>.

On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma <is...@juma.me.uk> wrote:

> Hi all,
>
> A few points:
>
> 1. I think the way backwards compatibility is being used here is not
> correct. Any functionality that is only enabled if set via a config is
> backwards compatible. People may disagree with the functionality or the
> config, but it's not a backwards compatibility issue.
>

We are talking about both broker and producer as a  single entity and run
by the same team/users. Allowing newer producer to create topics on a older
broker when auto.create.topics.enable set to false, breaks  server side
contract that this config offered from the beginning.  IMO, it clearly
isn't backward compatible. User who set auto.create.topic.enable on broker
will not be the same who will turn it on producer side .


> 2. It's an interesting question if auto topic creation via the producer
> should be a server driven choice or not. I can see the desire to have a
> server-driven default, but it seems like this is often application
> specific. Because the functionality is trivially available via AdminClient
> (released 2 years ago), it's not quite possible to control what
> applications do without the use of ACLs or policies today.
>
>
>
Producers & consumers are the majority of the clients in Kafka ecosystem.
Just because AdminClient shipped a while back that doesn't mean all users
adopting to it. To this day lot more users are aware of Producer & Consumer
APIs and running them in production compare to AdminClient.


> 3. Changing the create topics request in this way is highly unintuitive in
> my opinion and it relies on every client to pass the new field. For
> example, if librdkafka added auto create functionality by relying on their
> AdminClient, it would behave differently than what is proposed here.
> Forcing every client to implement this change when calling auto create from
> the producer specifically seems odd
>

I am not sure why its unintuitive , protocols change. We add or upgrade the
existing protocols all the time.


Thanks,
Harsha

.
>
> Ismael
>
> On Thu, Aug 8, 2019 at 11:51 AM Jun Rao <ju...@confluent.io> wrote:
>
> Hi, Justine,
>
> Thanks for the KIP. Overall, it seems to be a good improvement.
>
> However, I think Harsha's point seems reasonable. We had
> auto.create.topics.enable config on the broker to allow admins to disable
> topic creation from the producer/consumer clients before we had the
> security feature. The need for that config is reduced with the security
> feature, but may still be present since not all places have security
> enabled. It's true that a non-secured environment is vulnerable to some
> additional attacks, but producer/consumer are the most common way for a
> user to interact with the broker. So, keeping that config for backward
> compatibility could still be useful if it's not introducing too much effort
> or extra confusion.
>
>
> Here is a one potential alternative way that I was thinking. We add a new
> field in the CreateTopicRequest to indicate whether it's from the producer
> or not. If auto.create.topics.enable is false, CreateTopicRequest from the
> producer will be rejected. We probably don't need to introduce the new
> config (which seems a bit hard to explain) in the producer. Instead, the
> new producer always uses MetadataRequest with AllowAutoTopicCreation set to
> false to get the metadata and if the metadata is not present, send the new
> CreateTopicRequest
> (assuming the broker supports it) to try to create the topic
> automatically. Whether the creation is allowed or not will be determined by
> the broker. This will make the behavior backward compatible and we can
> still achieve the main goal of the KIP, which is not relying on
> MetadataRequest for topic creation. What do you think?
>
>
> Thanks,
>
> Jun
>
> On Thu, Aug 8, 2019 at 1:34 AM M. Manna <ma...@gmail.com> wrote:
>
> Hi,
>
>
> If I may, perhaps you could simplify everything by using only
> 'auto.create.topics.enable' as a value along with true. In other words,
>
>
> the
>
> public interfaces section should only have
>
> [true,auto.create.topics.enable,
>
> false].
>
> The reason for this is that auto.create.topics.enable is already known to
> users as a "Server-SIde" config. So all you are saying is
>
> a) To avoid day 1 impact, it will follow whatever
>
> auto.create.topics.enable
>
>
> value is set.
> b) False means - no client side topic creation
> c) True means client side topic creation.
>
>
> It saves creating 2 more new strings :). But not too expensive anyway.
>
> Also, when you deprecate auto.create.topics.enable - you must provide
> sufficient logic to ensure that things like rolling upgrade doesn't
> temporarily break anything. I apologise if you have already accounted for
> this, but wanted to mention since I didn't notice this on the KIP.
>
> Let me know how this sounds.
>
> Regards,
>
> On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jo...@confluent.io>
>
> wrote:
>
> Hi Harsha,
>
> I think my message may have gotten lost in all the others.
>
> Two of the goals of this KIP are to 1) allow auto-creation on specific
> clients when the broker default is false and 2) eventually replace the
> broker config.
>
> In order to accomplish these two goals, we need the producer to be able
>
> to
>
>
> create topics despite the broker config. (How can we replace a function
> when we rely on it?)
> I think at this point we have a fundamental disagreement in what we
>
>
> should
>
>
> allow the producer to do.
> In my previous message I mentioned a config that would allow for the
>
>
> broker
>
> to prevent producer auto-creation. (It would be disabled by default.)
>
> It
>
> would fix your issue for now, but could lead to more complications
>
> later.
>
>
> Thank you,
> Justine
>
>
> On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz>
>
> wrote:
>
> On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
>
> On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe <
>
> cmccabe@apache.org
>
> wrote:
>
> On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
>
>
> Hi Colin,
> "Hmm... I'm not sure I follow. Users don't have to build their own
> tooling, right? They can use any of the shell scripts that we've
>
>
> shipped
>
> in
>
> the last few releases. For example, if any of your users run it,
>
> this
>
> shell
>
> script will delete all of the topics from your non-security-enabled
> cluster:
>
>
> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> --bootstrap-server localhost:9092 --list 2>/dev/null
> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> --bootstrap-server localhost:9092 --delete
> --topic
>
>
> They will need to fill in the correct bootstrap servers list, of
>
> course,
>
> not localhost. This deletion script will work on some pretty old
>
> brokers,
>
> even back to the 0.10 releases. It seems a little odd to trust your
>
> users
>
> with this power, but not trust them to avoid changing a particular
> configuration key."
>
> The above will blocked by the server if we set delete.topic.enable
>
> to
>
> false and thats exactly what I am asking for.
>
> Hi Harsha,
>
> I was wondering if someone was going to bring up that configuration
>
> :)
>
> it's an interesting complication, but globally disabling topic
>
> deletion
>
> is
>
> not very practical for most use-cases.
>
> In any case, there are plenty of other bad things that users with
>
> full
>
> permissions can do that aren't blocked by any server configuration.
>
> For
>
> example, they can delete every record in every topic. I can write a
>
> script
>
> for that too, and there's no server configuration you can set to
>
> disable
>
> it. Or I could simply create hundreds of thousands of topics, until
>
> cluster
>
> performance becomes unacceptable (this will be even more of a
>
> problem
>
> if
>
> someone configured delete.topic.enable as false). Or publish bad
>
> data
>
> to
>
> every topic, etc. etc.
>
> The point I'm trying to make here is that you can't rely on these
>
> kind
>
> of
>
> server-side configurations for security. At most, they're a way to
>
> set
>
> up
>
> certain very simple policies. But the policies are so simple that
>
> they're
>
> hardly ever useful any more.
>
> For example, if the problem you want to solve is that you want a
>
> user
>
> to
>
> only be able to create 50 topics and not delete anyone else's
>
> topics,
>
> you
>
> can solve that with a CreateTopicsPolicy that limits the number of
>
> topics,
>
> and some ACLs. There's no combination of auto.create.topics.enable
>
> and
>
> delete.topic.enable that will help here.
>
> Hi Colin,
>
> Well you gave the example that a user can delete the
>
> topics
>
> just by running that script :).
>
> I understand there are open APIs in Kafka and can lead to rogue
>
> clients
>
> taking advantage of it without proper security in place.
>
> What I am asking so far in this thread is , this KIP is changing
>
> the
>
> producer behavior and its not backward compatible.
>
> The change is backwards compatible. The default will still be
>
> server-side
>
> topic auto-creation, just like now.
>
> You will have to specifically change the producer config to get the
>
> new
>
> behavior.
>
> I disagree. Today server can turn off the topic creation neither
>
> producer
>
> or consumer can create a topic. With this KIP , producer can create a
>
> topic
>
> by turning on client side config when server side config is turned
>
> off.
>
> We can still achieve
>
> the main goal of the KIP which is to change MetadataRequest
>
> creating
>
> topics and send a CreateTopicRequest from Producer and also keep
>
> the
>
> server
>
> side config to have precedence. This KIP originally written to
>
> have
>
> server
>
> side preference and there is not much explanation why it changed to
>
> have
>
> Producer side preference.
>
> Arguing that AdminClient can do that and so we are going to make
>
> Producer
>
> do the same doesn't make sense.
>
> "The downside is that if we wanted to check a server side
>
> configuration
>
> before sending the create topics request, the code would be more
>
> complex.
>
> The behavior would also not be consistent with how topic
>
> auto-creation
>
> is
>
> handled in Kafka Streams."
>
> I am not sure why you need to check server side configuration
>
> before
>
> sending create topics request. A user enables producer side config
>
> to
>
>
> create topics.
> Producer sends a request to the broker and if the broker has
> auto.topic.create.enable to true (default) it will allow creation
>
>
> of
>
> topics. If it set to false it returns error back to the client.
>
> auto.topic.create.enable has never affected CreateTopicsRequest. If
>
> you
>
> submit a CreateTopicsRequest and you are authorized, a topic will
>
> be
>
> created, regardless of what the value of auto.topic.create.enable
>
> is.
>
> This
>
> behavior goes back a long way, to about 2016 (Kafka 0.10.1, I
>
> think?)
>
> I don't see how this behavior will be different in Kafka streams.
>
> By
>
> default server allows the topic creation and with this KIP, It will
>
> only
>
> allow creation of topic when both producer and server side are
>
> turned
>
> on.
>
> Its exactly the same behavior in KIP-361.
>
> I suggest running a streams job as a test. It creates the topics it
>
> needs
>
> using AdminClient. The value of auto.topic.create.enable is not
>
> relevant.
>
> Whether it is true or false, the topics will still be created.
>
> "In general, it would be nice if we could keep the security and
>
> access
>
> control model simple and not introduce a lot of special cases and
> exceptions. Kafka has basically converged on using ACLs and
> CreateTopicPolicy classes to control who can create topics and
>
> where.
>
> Adding more knobs seems like a step backwards, especially when the
>
> proposed
>
> knobs don't work consistently across components, and don't provide
>
> true
>
> security." This is not about access control at all. Shipping sane
>
> defaults
>
> should be prioritized.
>
> I don't think the default is really the question here. I think
>
> everyone
>
> agrees that the default for client-side auto-topic creation should
>
> be
>
> off.
>
> My point goes back to KIP-361 why we kept the server side config to
>
> have
>
> preference. We can keep bringing back the discussion of security
>
> policies
>
> but a producer client overiding server side setting is not just
>
> security
>
> policy issue .
>
> "Producer client can override server side setting and not consumer
>
> and
>
> also producer can only override creation of topic but no the
>
> replication
>
> and partitions. " This doesn't make sense to me and why we want to
> introduce such a confusing behavior.
>
> The scenarios that you're describing (such as dealing with a poorly
> configured client that tries to create topics it should not) do
>
> seem
>
> to
>
> be
>
> about access control.
>
> We keep talking about CreateTopicPolicy and yet we don't have
>
> default
>
> one
>
> and asking every user of Kafka implement their own doesn't make
>
> sense
>
> here.
>
> The point of CreateTopicPolicy is exactly that you can implement
>
> your
>
> own,
>
> though. It's a way of customizing what the policy is in your
>
> specific
>
> cluster.
>
> I agree that most people don't want to write a custom policy. But
>
> most
>
> of
>
> those people are satisfied with just setting up ACLs to set up
>
> simple
>
> policies like this user can create topics, that user can't, etc.
>
> That's
>
> why
>
> I suggested adding support for ACLs to insecure clusters might be
>
> helpful.
>
> Also, just as a side note, wouldn't it would be impossible for us
>
> to
>
> specify a default CreateTopicsPolicy that did anything at this
>
> point
>
> anyway
>
> without breaking backwards compatibility? Maybe I'm
>
> misunderstanding
>
> the
>
> proposal.
>
> I am asking about exact behavior that we shipped for consumer side
>
> https:/
>
> / cwiki. apache. org/ confluence/ display/ KAFKA/
>
> KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
>
>
> (
> https://cwiki.apache.org/confluence/display/KAFKA/
>
>
> KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
>
> )
>
> I still didn't get any response why this behavior shouldn't be
>
> exactly
>
> like Kafka consumer and why do we want to have producer to
>
> overrider
>
> server
>
> side config and while not allowing consumer to do so. We are not
>
> even
>
> allowing the same contract and this will cause more confusion from
>
> the
>
> users standpoint.
>
> Hmm. The consumer already has a way to override the server side
> configuration. See KIP-361: Add Consumer Configuration to Disable
>
> Auto
>
> Topic Creation. This lets the consumer skip auto-creating topics,
>
> even
>
> if
>
> auto-creation is enabled on the broker.
>
> Yes that's what I am saying and it doesn't allow topic creation if
>
> the
>
> server side auto-creation is disabled and in consumer its enabled.
>
> I
>
> would like to see the exact behavior for producer as stated in
>
> KIP-361.
>
> To be fair, the KIP should probably discuss why we don't want to
>
> implement
>
> client-side topic creation in the consumer in "rejected
>
> alternatives."
>
> Maybe Justine can add more context here in the KIP.
>
> The last time we talked about this, the reasoning was that we
>
> wanted
>
> to
>
> eventually get rid of consumer-side auto-topic creation entirely,
>
> and
>
> so
>
> it
>
> wasn't worth implementing an improved way of doing it. Producer
>
> side
>
> auto
>
> topic-creation, in contrast, will probably stick around, although
>
> we'd
>
> like
>
> to deprecate and remove the problematic server-side mechanism over
>
> time.
>
> We should implement the producer side topic creation with proper
>
> create
>
> topic request and I am in favor of this KIP.
>
> All I am asking is don't make the producer to override server side
>
> config
>
> and keep it similar to KIP-361 just like consumer, it honors what
>
> set
>
> on
>
> server side which takes the precedence. Changing this behavior in producer
> is backward incompatible and will catch users by surprise.
>
> All arguments for enforcing producer side overriding config can
>
> still
>
> be
>
> achieved. Server side auto creation turned on by default and any
>
> new
>
> producer client setting their auto creation config to true will
>
> create
>
> topics in default behavior and any user who set server side to
>
> false
>
> and
>
> upgrades to latest kafka with this KIP will not see any unwanted
>
> behavior
>
> from clients. IMO this is not a unreasonable request on this KIP
>
> and
>
> this
>
> requested behavior is exactly what the KIP initially proposed.
>
> Thanks,
>
> Harsha
>
>
> best,
> Colin
>
>
> On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache.
>
> org (
>
> cmccabe@apache.org ) > wrote:
>
> On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
>
> Not sure how the AdminClient applies here, It is an external API
>
> and
>
> is
>
> not part of KafkaProducer so any user who updates to latest version
>
> of
>
>
> Kafka with this feature get to create the topics.
> They have to build a tooling around AdminClient allow themselves to
>
>
> create
>
> topics.
>
> Hi Harsha,
>
> Hmm... I'm not sure I follow. Users don't have to build their own
>
> tooling,
>
> right? They can use any of the shell scripts that we've shipped in
>
> the
>
> last
>
> few releases. For example, if any of your users run it, this shell
>
> script
>
> will delete all of the topics from your non-security-enabled
>
> cluster:
>
>
> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> --bootstrap-server localhost:9092 --list 2>/dev/null
> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> --bootstrap-server localhost:9092 --delete
> --topic
>
>
> They will need to fill in the correct bootstrap servers list, of
>
> course,
>
> not localhost. This deletion script will work on some pretty old
>
> brokers,
>
> even back to the 0.10 releases. It seems a little odd to trust your
>
> users
>
> with this power, but not trust them to avoid changing a particular
> configuration key.
>
> There is no behavior in Kafka producer that allowed users to delete
>
> the
>
> topics or delete the records. So citing them as an example doesn't
>
> makes
>
> sense in this context.
>
> I think Kafka Streams is relevant here. After all, it's software
>
> that
>
> we
>
> ship as part of the official Kafka release. And it auto-creates
>
> topics
>
> even
>
> when auto.create.topics.enable is set to false on the broker.
>
> I think that auto.create.topics.enable was never intended as a
>
> security
>
> setting to control access. It was intended as a way to disable the
> broker-side auto-create feature specifically, because there were
>
> some
>
> problems with that specific feature. Broker-side auto-creation is
> frustrating because it's on by default, and because it can
>
> auto-create
>
> topics even if the producer or consumer didn't explicitly ask for
>
> that.
>
> Neither of those problems applies to this KIP: producers have to
> specifically opt in, and it won't be on by default. Basically, we
>
> think
>
> that client-side auto-creation is actually a lot better-- hence
>
> this
>
> KIP
>
> :)
>
>
> But there is
> a functionality which allowed creation of topics if they don't
>
>
> exist
>
> in
>
> the
>
> cluster and this behavior could be controlled by a config on the
>
> server
>
> side. Now with this KIP we are allowing producer to make that
>
> decision
>
> without any gateway on the server via configs. Any user who is not
>
> aware
>
> of
>
>
> such changes
> can accidentally create these topics and we are essentially
>
>
> removing
>
> a
>
> config that exists in brokers today to block this accidental
>
> creation
>
> and
>
> allowing clients to take control.
>
> Again, I hope I'm not misinterpreting, but I don't see how this can
>
> be
>
> turned on accidentally. The user would have to specifically turn
>
> this
>
> on
>
> in
>
> the producer by setting the configuration key.
>
> I still didn't get any positives of not having server side configs?
>
> if
>
> you
>
> want to turn them on and allow any client to create topics set the
>
> default
>
>
> to true
> and allow users who doesn't want to have this feature let them turn
>
>
> them
>
> off. It will be the exact behavior as it is today , as far as
>
> producer
>
> is
>
>
> concerned. I am not
> understanding why not having server side configs to gateways such a
>
>
> hard
>
> requirement and this behavior exists today. As far I am concerned
>
> this
>
> breaks the backward compatibility.
>
> The downside is that if we wanted to check a server side
>
> configuration
>
> before sending the create topics request, the code would be more
>
> complex.
>
> The behavior would also not be consistent with how topic
>
> auto-creation
>
> is
>
> handled in Kafka Streams.
>
> In general, it would be nice if we could keep the security and
>
> access
>
> control model simple and not introduce a lot of special cases and
> exceptions. Kafka has basically converged on using ACLs and
> CreateTopicPolicy classes to control who can create topics and
>
> where.
>
> Adding more knobs seems like a step backwards, especially when the
>
> proposed
>
> knobs don't work consistently across components, and don't provide
>
> true
>
> security.
>
> Maybe we should support an insecure mode where there are still
>
> users
>
> and
>
> ACLs. That would help people who don't want to set up Kerberos,
>
> etc.
>
> but
>
> still want to protect against misconfigured clients or accidents.
>
> Hadoop
>
> has something like this, and I think it was useful. NFS also
>
> supported
>
> (supports?) a mode where you just pass whatever user ID you want
>
> and
>
> the
>
> system believes you. These things clearly don't protect against
>
> malicious
>
> users, but they can help set up policies when needed.
>
>
> best,
> Colin
>
>
>
> Thanks,
> Harsha
>
>
> On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache.
>
> org (
>
> cmccabe@apache.org ) > wrote:
>
> Hi Harsha,
>
> Thanks for taking a look.
>
> I'm not sure I follow the discussion about AdminClient.
>
> KafkaAdminClient
>
> has been around for about 2 years at this point as a public class.
>
> There
>
> are many programs that use it to automatically create topics. Kafka
> Streams does this, for example. If any of your users run Kafka
>
> Streams,
>
> they will be auto-creating topics, regardless of what setting you
>
> use
>
> for
>
> auto.create.topics.enable.
>
> A big part of the annoyance of auto-topic creation right now is
>
> that
>
> it is
>
> on by default. The new configuration proposed by KIP-487 wouldn't
>
> be.
>
> Users would have to explicitly opt in to the new behavior of
>
> client-side
>
> topic creation. If you run without security, you're already
>
> putting a
>
> huge
>
> amount of trust in your users. For example, you trust them not to
>
> delete
>
> records with the kafka-delete-records. sh (
>
> http://kafka-delete-records.
>
>
> sh/
> ) command, or delete topics with kafka-topics. sh (
>
>
> http://kafka-topics.
>
>
> sh/
> ). Trusting them not to set a certain config value seems minor in
> comparison, right?
>
>
>
> best,
> Colin
>
>
> On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
>
>
> Hi,
> Even with policies one needs to implement that, so for every
>
>
> user who
>
> doesn't want a producer to create topics or have limits around
>
> partitions &
>
> replication factor they have to implement a policy. The KIP is
>
> changing
>
> the behavior , it might not be introducing
>
> the
>
> new functionality but it will enable producers to override the
>
> create
>
> topic
>
> config settings on the broker. What I am asking for to provide a
>
> config
>
> that will disable auto creation of topics and if its enabled set
>
> some
>
> sane
>
> defaults so that clients can create a topic with in those limits. I
>
> don't
>
>
> see how this not related to this KIP.
> If the server config options as I suggested doesn't interest
>
>
> you at
>
>
> least have a default CreateTopicPolices in place.
> To give an example, In our environment we disable the
> auto.create.topic.enable and force users to go through a
>
>
> centralized
>
> service as we want capture more details about the topic creation
>
> and
>
> requirements. With this KIP, a producer can create a topic with no
>
> bounds.
>
> Another example max.message.size we define that at cluster level
>
> and one
>
> can override max.messsage.size at topic level, any misconfiguration
>
> at
>
> this
>
> will cause service degradation. Its not always about the rogue
>
> clients,
>
> users can easily misconfigure and can cause an outage. Again we can
>
> talk
>
> about CreateTopicPolicy but without having a
>
> default
>
> implementation and asking everyone to implement their own while
>
> changing
>
> the behavior in producer doesn't make sense to me.
>
>
> Thanks,
> Harsha
>
>
> On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me.
>
> uk (
>
> ismael@juma.me.uk ) >
>
> wrote:
>
> Hi Harsha,
>
> I mentioned policies and the authorizer. For example, with
> CreateTopicPolicy, you can implement the limits you describe. If
>
> you
>
> have
>
> ideas of how that should be improved, please submit a KIP. My
>
> point is
>
> that
>
> this KIP is not introducing any new functionality with regards to
>
> what
>
> rogue clients can do. It's using the existing protocol that is
>
> already
>
> exposed via the AdminClient. So, I don't think we need to address
>
> it in
>
> this KIP. Does that make sense?
>
> Ismael
>
> On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
>
> kafka@ harsha. io ( kafka@harsha.io ) >
>
> wrote:
>
>
> Ismael,
> Sure AdminClient can do that and we should've shipped a config or
>
>
> use
>
> the
>
> existing one to block that. Not all users are yet to upgrade to
>
> AdminClient
>
> and start using that to cause issues yet. In shared environment we
>
> should
>
> allow server to set sane defaults and not allow every client to go
>
> ahead
>
> create random no.of topic/partitions and replication factor. Even
>
> if
>
> the
>
> users want to allow topic creation proposed in the KIP , it makes
>
> sense to
>
> have some guards against the no.of partitions and replication
>
> factor.
>
> Authorizer is not always an answer to block requests and having
>
> users
>
> set
>
> server side configs to protect a multi-tenant environment is
>
> required.
>
> In a
>
> non-secure environment Authorizer is a blunt instrument either you
>
> end
>
> up
>
>
> blocking everyone or allowing everyone.
> I am asking to have server side that allow clients to create
>
>
> topics or
>
> not
>
> , if they are allowed set a ceiling on max no.of partitions and
> replication-factor.
>
> -Harsha
>
> On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com (
>
> ismaelj@gmail.com
>
> )
>
> wrote:
>
> Harsha,
>
> Rogue clients can use the admin client to create topics and
>
> partitions.
>
> ACLs and policies can help in that case as well as this one.
>
> Ismael
>
> On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha.
>
> io
>
> (
>
> kafka@harsha.io ) >
>
> wrote:
>
>
> Hi Justine,
> Thanks for the KIP.
> "When server-side auto-creation is disabled, client-side
>
>
> auto-creation
>
>
> will try to use client-side configurations"
> If I understand correctly, this KIP is removing any server-side
>
>
> blocking
>
>
> client auto creation of topic?
> if so this will present potential issue of rogue client creating
>
>
> ton of
>
> topic-partitions and potentially bringing down the service for
>
> everyone
>
> or
>
>
> degrade the service itself.
> By reading the KIP its not clear to me that there is a clear way to
>
>
> block
>
> auto creation topics of all together from clients by server side
>
> config.
>
> Server side configs of default topic, partitions should take higher
> precedence and client shouldn't be able to create a topic with
>
> higher
>
> no.of
>
> partitions, replication than what server config specifies.
>
>
> Thanks,
> Harsha
>
>
> On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
>
> Hi all,
> I made some changes to the KIP. Hopefully this configuration change
>
>
> will
>
> make things a little clearer.
>
> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
>
> https://cwiki.
>
>
> apache.org/confluence/display/KAFKA/ )
> KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
>
>
> Please let me know if you have any feedback or questions!
>
>
> Thank you,
> Justine
>
>
> On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache.
>
> org (
>
> cmccabe@apache.org ) >
>
> wrote:
>
> Hi Mickael,
>
> I think you bring up a good point. It would be better if we didn't
>
> ever
>
> have to set up client-side configuration for this feature, and
>
> KIP-464
>
> would let us skip this entirely.
>
> On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
>
>
> Hi Mickael,
> I agree that KIP-464 works on newer brokers, but I was a bit
>
>
> worried
>
> how
>
> things would play out on older brokers that* do not *have KIP 464
>
> included.
>
> Is it enough to throw an error in this case when producer configs
>
> are
>
> not
>
> specified?
>
> I think the right thing to do would be to log an error message in
>
> the
>
> client. We will need to have that capability in any case, to cover
> scenarios like the client trying to auto-create a topic that they
>
> don't
>
> have permission to create. Or a client trying to create a topic on
>
> a
>
> broker
>
> so old that CreateTopicsRequest is not supported.
>
> The big downside to relying on KIP-464 is that it is a very recent
>
> feature
>
> -- so recent that it hasn't even made its way to any official
>
> Apache
>
> release. It's scheduled for the upcoming 2.4 release in a few
>
> months.
>
> So if you view this KIP as a step towards removing broker-side
> auto-create, you might want to support older brokers just to
>
> accelerate
>
> adoption, and hasten the day when we can finally flip broker-side
> auto-create to off (or even remove it entirely).
>
> I have to agree, though, that having client-side configurations for
>
> number
>
> of partitions and replication factor is messy. Maybe it would be
>
> worth
>
> it
>
> to restrict support to post-KIP-464 brokers, if we could avoid
>
> creating
>
> more configs.
>
>
> best,
> Colin
>
>
> On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
>
> mickael. maison@ gmail. com ( mickael.maison@gmail.com )
>
> wrote:
>
> Hi Justine,
>
> We can rely on KIP-464 which allows to omit the partition count or
> replication factor when creating a topic. In that case, the broker
>
> defaults
>
> are used.
>
> On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
>
> Michael,
> That makes sense to me!
> To clarify, in the current state of the KIP, the producer does not
>
>
> rely
>
> on
>
> the broker to autocreate--if the broker's config is disabled, then
>
> the
>
> producer can autocreate on its own with a create topic request (the
>
> same
>
>
> type of request the admin client uses).
> However, if both configs are enabled, the broker will autocreate
>
>
> through
>
> a
>
> metadata request before the producer gets a chance. Of course, the
>
> way
>
> to
>
> avoid this, is to do as you suggested, and set
>
> the
>
> "allow_auto_topic_creation" field to false.
>
> I think the only thing we need to be careful with in this setup is
>
> without
>
> KIP 464, we can not use broker defaults for this topic. A user
>
> needs
>
> to
>
> specify the number of partition and replication factor in the
>
> config.
>
> An
>
> alternative to this is to have coded defaults for when these
>
> configs
>
> are
>
> unspecified, but it is not immediately apparent what these defaults
>
> should
>
> be.
>
>
> Thanks again for reading my KIP,
> Justine
>
>
> On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
>
> mickael. maison@ gmail. com ( mickael.maison@gmail.com )
>
> wrote:
>
> Hi Justine,
>
>
> Thanks for the response!
> In my opinion, it would be better if the producer did not rely at
>
>
> all
>
>
> on the broker auto create feature as this is what we're aiming to
> deprecate. When requesting metadata we can set the
> "allow_auto_topic_creation" field to false to avoid the broker auto
> creation. Then if the topic is not existing, send a
>
>
> CreateTopicRequest.
>
> What do you think?
>
> On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Currently the way it is implemented, the broker auto-creation
>
> configuration
>
> takes precedence. The producer will not use the CreateTopics
>
> request.
>
> (Technically it can--but the topic will already be created
>
> through
>
> the
>
> broker, so it will never try to create the topic.) It is possible
>
> to
>
> change this however, and I'd be happy to
>
> discuss
>
> the
>
> benefits of this alternative.
>
>
> Thank you,
> Justine
>
>
> On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
>
> mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP!
>
> In case auto creation is enabled on both the client and server,
>
> will
>
> the producer still use the AdminClient (CreateTopics request)
>
> to
>
> create topics? and not rely on the broker auto create. I'm
>
> guessing the
>
> answer is yes but can you make it explicit.
>
> Thank you
>
> On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
>
> Hi,
> Just a friendly reminder to take a look at this KIP if you
>
>
> have
>
> the
>
> time.
>
> I was thinking about broker vs. client default precedence,
>
> and I
>
> think it
>
> makes sense to keep the broker as the default used when both
>
> client-side
>
> and broker-side defaults are configured. The idea is that
>
> there
>
> would be
>
> pretty clear documentation, and that many systems with
>
> configurations
>
> that
>
> the client could not change would likely have the auto-create
>
> default
>
> off.
>
> (In cloud for example).
>
>
> It also seems like in most cases, the consumer config
> ' allow. auto. create. topics ( http://allow.auto.create.topics/
>
>
> ) '
>
> was
>
> created to actually prevent
>
> the
>
> creation
>
> of
>
> topics, so the loss of creation functionality will not be a
>
> big
>
> problem.
>
> I'm happy to discuss any other compatibility problems or
>
> components
>
> of
>
> this KIP.
>
>
> Thank you,
> Justine
>
>
> On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io )
>
> wrote:
>
> Hello all,
>
> I was looking at this KIP again, and there is a decision I
>
> made
>
> that I
>
> think is worth discussing.
>
>
> In the case where both the broker and producer's
> 'auto.create.topics.enable' are set to true, we have to
>
>
> choose
>
> either
>
> the
>
> broker configs or the producer configs for the replication
> factor/partitions.
>
> Currently, the decision is to have the broker defaults take
>
> precedence.
>
> (It is easier to do this in the implementation.) It also
>
> makes
>
> some
>
> sense
>
> for this behavior to take precedence since this behavior
>
> already
>
> occurs as
>
> the default.
>
> However, I was wondering if it would be odd for those who
>
> can
>
> only
>
> see
>
> the
>
> producer side to set configs for replication
>
> factor/partitions
>
> and
>
> see
>
> different results. Currently the documentation for the
>
> config
>
> states
>
> that
>
> the config values are only used when the broker config is
>
> not
>
> enabled,
>
> but
>
> this might not always be clear to the user. Changing the
>
> code
>
> to
>
> have
>
> the
>
> producer's configurations take precedence is possible, but
>
> I
>
> was
>
> wondering
>
> what everyone thought.
>
>
> Thank you,
> Justine
>
>
> On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Just a quick update--
>
> It seems that enabling both the broker and producer
>
> configs
>
> works
>
> fine,
>
> except that the broker configurations for partitions,
>
> replication
>
> factor
>
>
> take precedence.
> I don't know if that is something we would want to
>
>
> change, but
>
> I'll be
>
> updating the KIP for now to reflect this. Perhaps we would
>
> want to
>
> add more
>
> to the documentation of the the producer configs to
>
> clarify.
>
>
> Thank you,
> Justine
>
>
> On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Hi Colin,
>
> Thanks for looking at the KIP. I can definitely add to
>
> the
>
> title
>
> to
>
> make
>
> it more clear.
>
> It makes sense that both configurations could be turned
>
> on
>
> since
>
> there
>
> are many cases where the user can not control the
>
> server-side
>
> configurations. I was a little concerned about how both
>
> interacting
>
> would
>
> work out -- if there would be to many requests for new
>
> topics,
>
> for
>
> example.
>
> But it since it does make sense to allow both
>
> configurations
>
> enabled, I
>
> will test out some scenarios and I'll change the KIP to
>
> support
>
> this.
>
> I also agree with documentation about distinguishing the
>
> differences. I
>
> was having some trouble with the wording but I like the
>
> phrases
>
> "server-side" and "client-side." That's a good
>
> distinction I
>
> can
>
> use
>
> when
>
> describing.
>
> I'll try to update the KIP soon keeping everyone's input
>
> in
>
> mind.
>
>
> Thanks,
> Justine
>
>
> On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
>
> cmccabe@ apache. org ( cmccabe@apache.org )
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP. This seems like a good step towards
>
> removing
>
> server-side topic auto-creation.
>
> We should add included "client-side" to the title of
>
> the KIP
>
> somewhere,
>
> to make it clear that we're talking about client-side
>
> auto
>
> creation.
>
> The KIP says:
>
> In order to automatically create topics with the
>
> producer, the
>
> producer's
>
> auto.create.topics.enable config must be set to true
>
> and
>
> the
>
> broker
>
> config should be set to false
>
> From a user's point of view, this seems
>
> counter-intuitive.
>
> In
>
> order to
>
> auto-create topics the broker's
>
> auto.create.topics.enable
>
> config
>
> should be
>
> set to false? It seems like the server-side
>
> auto-create is
>
> unrelated to
>
> the client-side auto-create. We could have both turned
>
> on
>
> (and
>
> I'm
>
> sure
>
> that in the real world, people will try this
>
> configuration...)
>
> There's no
>
> reason not to support this, I think.
>
> We should add some documentation explaining the
>
> difference
>
> between
>
> server-side and client-side auto-creation. Without
>
> documentation,
>
> an admin
>
> might think that they had disabled all forms of
>
> auto-creation by
>
> setting
>
> the -side setting to false-- but this is not the case,
>
> of
>
> course.
>
>
> best,
> Colin
>
>
> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
>
> Hi Dhruvil,
>
>
> Thanks for reading the KIP!
> That was the general idea for deprecation. We would
>
>
> log a
>
> warning
>
> when the
>
>
> config is enabled on the broker.
> I also believe that there would be a change to
>
>
> documentation.
>
> If there is anything else that should be done, please
>
> let
>
> me
>
> know!
>
> Justine
>
> On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
>
> dhruvil@ confluent. io ( dhruvil@confluent.io ) >
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP, this is great!
>
> Could you add some more information about what
>
> deprecating
>
> the
>
> broker
>
> configuration means? Would we log a warning in the
>
> logs
>
> when
>
> auto
>
> topic
>
> creation is enabled on the broker, for example?
>
>
> Thanks,
> Dhruvil
>
>
> On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Hello all,
>
> I'd like to start a discussion thread for KIP-487. This KIP plans
>
> to
>
> deprecate the current system of
>
> auto-creating
>
> topics
>
> through requests to the metadata and give the
>
> producer the
>
> ability to
>
> automatically create topics instead.
>
> More information can be found here:
>
> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
>
> https://cwiki.
>
>
> apache.org/confluence/display/KAFKA/ )
> KIP-487%3A+Automatic+Topic+Creation+on+Producer
>
>
>
> Thank you,
> Justine Olshan
>
>
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Ismael Juma <is...@juma.me.uk>.

Hi all,

A few points:

1. I think the way backwards compatibility is being used here is not
correct. Any functionality that is only enabled if set via a config is
backwards compatible. People may disagree with the functionality or the
config, but it's not a backwards compatibility issue.

2. It's an interesting question if auto topic creation via the producer
should be a server driven choice or not. I can see the desire to have a
server-driven default, but it seems like this is often application
specific. Because the functionality is trivially available via AdminClient
(released 2 years ago), it's not quite possible to control what
applications do without the use of ACLs or policies today.

3. Changing the create topics request in this way is highly unintuitive in
my opinion and it relies on every client to pass the new field. For
example, if librdkafka added auto create functionality by relying on their
AdminClient, it would behave differently than what is proposed here.
Forcing every client to implement this change when calling auto create from
the producer specifically seems odd.

Ismael

On Thu, Aug 8, 2019 at 11:51 AM Jun Rao <ju...@confluent.io> wrote:

> Hi, Justine,
>
> Thanks for the KIP. Overall, it seems to be a good improvement.
>
> However, I think Harsha's point seems reasonable. We had
> auto.create.topics.enable config on the broker to allow admins to disable
> topic creation from the producer/consumer clients before we had the
> security feature. The need for that config is reduced with the security
> feature, but may still be present since not all places have security
> enabled. It's true that a non-secured environment is vulnerable to some
> additional attacks, but producer/consumer are the most common way for a
> user to interact with the broker. So, keeping that config for backward
> compatibility could still be useful if it's not introducing too much effort
> or extra confusion.
>
> Here is a one potential alternative way that I was thinking. We add a new
> field in the CreateTopicRequest to indicate whether it's from the producer
> or not. If auto.create.topics.enable is false, CreateTopicRequest from the
> producer will be rejected. We probably don't need to introduce the new
> config (which seems a bit hard to explain) in the producer. Instead, the
> new producer always uses MetadataRequest with AllowAutoTopicCreation set to
> false to get the metadata and if the metadata is not present, send the
> new CreateTopicRequest
> (assuming the broker supports it) to try to create the topic automatically.
> Whether the creation is allowed or not will be determined by the broker.
> This will make the behavior backward compatible and we can still achieve
> the main goal of the KIP, which is not relying on MetadataRequest for topic
> creation. What do you think?
>
> Thanks,
>
> Jun
>
> On Thu, Aug 8, 2019 at 1:34 AM M. Manna <ma...@gmail.com> wrote:
>
> > Hi,
> >
> > If I may, perhaps you could simplify everything by using only
> > 'auto.create.topics.enable' as a value along with true. In other words,
> the
> > public interfaces section should only have
> [true,auto.create.topics.enable,
> > false].
> >
> > The reason for this is that auto.create.topics.enable is already known to
> > users as a "Server-SIde" config. So all you are saying is
> >
> > a) To avoid day 1 impact, it will follow whatever
> auto.create.topics.enable
> > value is set.
> > b) False means - no client side topic creation
> > c) True means client side topic creation.
> >
> > It saves creating 2 more new strings :). But not too expensive anyway.
> >
> > Also, when you deprecate auto.create.topics.enable - you must provide
> > sufficient logic to ensure that things like rolling upgrade doesn't
> > temporarily break anything. I apologise if you have already accounted for
> > this, but wanted to mention since I didn't notice this on the KIP.
> >
> > Let me know how this sounds.
> >
> > Regards,
> >
> > On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jo...@confluent.io>
> wrote:
> >
> > > Hi Harsha,
> > >
> > > I think my message may have gotten lost in all the others.
> > >
> > > Two of the goals of this KIP are to 1) allow auto-creation on specific
> > > clients when the broker default is false and 2) eventually replace the
> > > broker config.
> > >
> > > In order to accomplish these two goals, we need the producer to be able
> > to
> > > create topics despite the broker config. (How can we replace a function
> > > when we rely on it?)
> > > I think at this point we have a fundamental disagreement in what we
> > should
> > > allow the producer to do.
> > > In my previous message I mentioned a config that would allow for the
> > broker
> > > to prevent producer auto-creation. (It would be disabled by default.)
> It
> > > would fix your issue for now, but could lead to more complications
> later.
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz>
> > wrote:
> > > >
> > > > > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> > > > >
> > > > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe <
> cmccabe@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> > > > >
> > > > > Hi Colin,
> > > > > "Hmm... I'm not sure I follow. Users don't have to build their own
> > > > > tooling, right? They can use any of the shell scripts that we've
> > > shipped
> > > > in
> > > > > the last few releases. For example, if any of your users run it,
> this
> > > > shell
> > > > > script will delete all of the topics from your non-security-enabled
> > > > > cluster:
> > > > >
> > > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > > --bootstrap-server localhost:9092 --delete
> > > > > --topic
> > > > >
> > > > > They will need to fill in the correct bootstrap servers list, of
> > > course,
> > > > > not localhost. This deletion script will work on some pretty old
> > > brokers,
> > > > > even back to the 0.10 releases. It seems a little odd to trust your
> > > users
> > > > > with this power, but not trust them to avoid changing a particular
> > > > > configuration key."
> > > > >
> > > > > The above will blocked by the server if we set delete.topic.enable
> to
> > > > > false and thats exactly what I am asking for.
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > I was wondering if someone was going to bring up that configuration
> > :)
> > > > >
> > > > > it's an interesting complication, but globally disabling topic
> > deletion
> > > > is
> > > > > not very practical for most use-cases.
> > > > >
> > > > > In any case, there are plenty of other bad things that users with
> > full
> > > > > permissions can do that aren't blocked by any server configuration.
> > For
> > > > > example, they can delete every record in every topic. I can write a
> > > > script
> > > > > for that too, and there's no server configuration you can set to
> > > disable
> > > > > it. Or I could simply create hundreds of thousands of topics, until
> > > > cluster
> > > > > performance becomes unacceptable (this will be even more of a
> problem
> > > if
> > > > > someone configured delete.topic.enable as false). Or publish bad
> data
> > > to
> > > > > every topic, etc. etc.
> > > > >
> > > > > The point I'm trying to make here is that you can't rely on these
> > kind
> > > of
> > > > > server-side configurations for security. At most, they're a way to
> > set
> > > up
> > > > > certain very simple policies. But the policies are so simple that
> > > they're
> > > > > hardly ever useful any more.
> > > > >
> > > > > For example, if the problem you want to solve is that you want a
> user
> > > to
> > > > > only be able to create 50 topics and not delete anyone else's
> topics,
> > > you
> > > > > can solve that with a CreateTopicsPolicy that limits the number of
> > > > topics,
> > > > > and some ACLs. There's no combination of auto.create.topics.enable
> > and
> > > > > delete.topic.enable that will help here.
> > > > >
> > > > > Hi Colin,
> > > > >
> > > > >             Well you gave the example that a user can delete the
> > topics
> > > > > just by running that script  :).
> > > > >
> > > > > I understand there are open APIs in Kafka and can lead to rogue
> > clients
> > > > > taking advantage of it without proper security in place.
> > > > >
> > > > > What I am asking so far in this thread is , this KIP is changing
> the
> > > > > producer behavior and its not backward compatible.
> > > > >
> > > > > The change is backwards compatible. The default will still be
> > > server-side
> > > > > topic auto-creation, just like now.
> > > > >
> > > > You will have to specifically change the producer config to get the
> new
> > > > > behavior.
> > > > >
> > > >
> > > >
> > > > I disagree.  Today server can turn off the topic creation  neither
> > > producer
> > > > or consumer can create a topic. With this KIP , producer can create a
> > > topic
> > > > by turning on client side config when server side config is turned
> off.
> > > >
> > > >
> > > > We can still achieve
> > > > > the main goal of the KIP which is to change MetadataRequest
> creating
> > > > > topics and send a CreateTopicRequest from Producer and also keep
> the
> > > > server
> > > > > side config to have precedence.  This KIP originally written to
> have
> > > > server
> > > > > side preference and there is not much explanation why it changed to
> > > have
> > > > > Producer side preference.
> > > > >
> > > > > Arguing that AdminClient can do that and so we are going to make
> > > Producer
> > > > > do the same doesn't make sense.
> > > > >
> > > > > "The downside is that if we wanted to check a server side
> > configuration
> > > > > before sending the create topics request, the code would be more
> > > complex.
> > > > > The behavior would also not be consistent with how topic
> > auto-creation
> > > is
> > > > > handled in Kafka Streams."
> > > > >
> > > > > I am not sure why you need to check server side configuration
> before
> > > > > sending create topics request. A user enables producer side config
> to
> > > > > create topics.
> > > > > Producer sends a request to the broker and if the broker has
> > > > > auto.topic.create.enable to true (default) it will allow creation
> of
> > > > > topics. If it set to false it returns error back to the client.
> > > > >
> > > > > auto.topic.create.enable has never affected CreateTopicsRequest. If
> > you
> > > > > submit a CreateTopicsRequest and you are authorized, a topic will
> be
> > > > > created, regardless of what the value of auto.topic.create.enable
> is.
> > > > This
> > > > > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I
> think?)
> > > > >
> > > > > I don't see how this behavior will be different in Kafka streams.
> By
> > > > > default server allows the topic creation and with this KIP, It will
> > > only
> > > > > allow creation of topic when both producer and server side are
> turned
> > > on.
> > > > > Its exactly the same behavior in KIP-361.
> > > > >
> > > > > I suggest running a streams job as a test. It creates the topics it
> > > needs
> > > > > using AdminClient. The value of auto.topic.create.enable is not
> > > relevant.
> > > > > Whether it is true or false, the topics will still be created.
> > > > >
> > > > > "In general, it would be nice if we could keep the security and
> > access
> > > > > control model simple and not introduce a lot of special cases and
> > > > > exceptions. Kafka has basically converged on using ACLs and
> > > > > CreateTopicPolicy classes to control who can create topics and
> where.
> > > > > Adding more knobs seems like a step backwards, especially when the
> > > > proposed
> > > > > knobs don't work consistently across components, and don't provide
> > true
> > > > > security." This is not about access control at all. Shipping sane
> > > > defaults
> > > > > should be prioritized.
> > > > >
> > > > > I don't think the default is really the question here. I think
> > everyone
> > > > > agrees that the default for client-side auto-topic creation should
> be
> > > > off.
> > > > >
> > > > > My point goes back to KIP-361 why we kept the server side config to
> > > have
> > > > > preference. We can keep bringing back the discussion of security
> > > policies
> > > > > but a producer client overiding server side setting is not just
> > > security
> > > > > policy issue .
> > > > >
> > > > > "Producer client can override server side setting and not consumer
> > and
> > > > > also producer can only override creation of topic but no the
> > > replication
> > > > > and partitions. " This doesn't make sense to me and why we want to
> > > > > introduce such a confusing behavior.
> > > > >
> > > > > The scenarios that you're describing (such as dealing with a poorly
> > > > > configured client that tries to create topics it should not) do
> seem
> > to
> > > > be
> > > > > about access control.
> > > > >
> > > > > We keep talking about CreateTopicPolicy and yet we don't have
> default
> > > one
> > > > > and asking every user of Kafka implement their own doesn't make
> sense
> > > > here.
> > > > >
> > > > > The point of CreateTopicPolicy is exactly that you can implement
> your
> > > > own,
> > > > > though. It's a way of customizing what the policy is in your
> specific
> > > > > cluster.
> > > > >
> > > > > I agree that most people don't want to write a custom policy. But
> > most
> > > of
> > > > > those people are satisfied with just setting up ACLs to set up
> simple
> > > > > policies like this user can create topics, that user can't, etc.
> > That's
> > > > why
> > > > > I suggested adding support for ACLs to insecure clusters might be
> > > > helpful.
> > > > >
> > > > > Also, just as a side note, wouldn't it would be impossible for us
> to
> > > > > specify a default CreateTopicsPolicy that did anything at this
> point
> > > > anyway
> > > > > without breaking backwards compatibility? Maybe I'm
> misunderstanding
> > > the
> > > > > proposal.
> > > > >
> > > > > I am asking about exact behavior that we shipped for consumer side
> > > > https:/
> > > > > / cwiki. apache. org/ confluence/ display/ KAFKA/
> > > > >
> KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > > > (
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > >
> KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > > > )
> > > > >
> > > > > I still didn't get any response why this behavior shouldn't be
> > exactly
> > > > > like Kafka consumer and why do we want to have producer to
> overrider
> > > > server
> > > > > side config and while not allowing consumer to do so. We are not
> even
> > > > > allowing the same contract and this will cause more confusion from
> > the
> > > > > users standpoint.
> > > > >
> > > > > Hmm. The consumer already has a way to override the server side
> > > > > configuration. See KIP-361: Add Consumer Configuration to Disable
> > Auto
> > > > > Topic Creation. This lets the consumer skip auto-creating topics,
> > even
> > > if
> > > > > auto-creation is enabled on the broker.
> > > > >
> > > > > Yes that's what I am saying and it doesn't allow topic creation if
> > the
> > > > > server side auto-creation is disabled and in consumer its enabled.
> >  I
> > > > > would like to see the exact behavior for producer as stated in
> > KIP-361.
> > > > >
> > > > > To be fair, the KIP should probably discuss why we don't want to
> > > > implement
> > > > > client-side topic creation in the consumer in "rejected
> > alternatives."
> > > > > Maybe Justine can add more context here in the KIP.
> > > > >
> > > > > The last time we talked about this, the reasoning was that we
> wanted
> > to
> > > > > eventually get rid of consumer-side auto-topic creation entirely,
> and
> > > so
> > > > it
> > > > > wasn't worth implementing an improved way of doing it. Producer
> side
> > > auto
> > > > > topic-creation, in contrast, will probably stick around, although
> > we'd
> > > > like
> > > > > to deprecate and remove the problematic server-side mechanism over
> > > time.
> > > > >
> > > > > We should implement the producer side topic creation with proper
> > create
> > > > > topic request  and I am in favor of this KIP.
> > > > >
> > > > > All I am asking is don't make the producer to override server side
> > > config
> > > > > and keep it similar to KIP-361 just like consumer, it honors what
> set
> > > on
> > > > > server side which takes the precedence.   Changing this behavior in
> > > > > producer is backward incompatible and will catch users by surprise.
> > > > >
> > > > > All arguments for enforcing producer side overriding config can
> still
> > > be
> > > > > achieved. Server side auto creation turned on by default and  any
> new
> > > > > producer client setting their auto creation config to true will
> > create
> > > > > topics in default behavior and any user who set server side to
> false
> > > and
> > > > > upgrades to latest kafka with this KIP will not see any unwanted
> > > behavior
> > > > > from clients.  IMO this is not a unreasonable request on this KIP
> and
> > > > this
> > > > > requested behavior is exactly what the KIP initially proposed.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Harsha
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache.
> org (
> > > > > cmccabe@apache.org ) > wrote:
> > > > >
> > > > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > > > >
> > > > > Not sure how the AdminClient applies here, It is an external API
> and
> > is
> > > > > not part of KafkaProducer so any user who updates to latest version
> > of
> > > > > Kafka with this feature get to create the topics.
> > > > > They have to build a tooling around AdminClient allow themselves to
> > > > >
> > > > > create
> > > > >
> > > > > topics.
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > Hmm... I'm not sure I follow. Users don't have to build their own
> > > > tooling,
> > > > > right? They can use any of the shell scripts that we've shipped in
> > the
> > > > last
> > > > > few releases. For example, if any of your users run it, this shell
> > > script
> > > > > will delete all of the topics from your non-security-enabled
> cluster:
> > > > >
> > > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > > --bootstrap-server localhost:9092 --delete
> > > > > --topic
> > > > >
> > > > > They will need to fill in the correct bootstrap servers list, of
> > > course,
> > > > > not localhost. This deletion script will work on some pretty old
> > > brokers,
> > > > > even back to the 0.10 releases. It seems a little odd to trust your
> > > users
> > > > > with this power, but not trust them to avoid changing a particular
> > > > > configuration key.
> > > > >
> > > > > There is no behavior in Kafka producer that allowed users to delete
> > the
> > > > > topics or delete the records. So citing them as an example doesn't
> > > makes
> > > > > sense in this context.
> > > > >
> > > > > I think Kafka Streams is relevant here. After all, it's software
> that
> > > we
> > > > > ship as part of the official Kafka release. And it auto-creates
> > topics
> > > > even
> > > > > when auto.create.topics.enable is set to false on the broker.
> > > > >
> > > > > I think that auto.create.topics.enable was never intended as a
> > security
> > > > > setting to control access. It was intended as a way to disable the
> > > > > broker-side auto-create feature specifically, because there were
> some
> > > > > problems with that specific feature. Broker-side auto-creation is
> > > > > frustrating because it's on by default, and because it can
> > auto-create
> > > > > topics even if the producer or consumer didn't explicitly ask for
> > that.
> > > > > Neither of those problems applies to this KIP: producers have to
> > > > > specifically opt in, and it won't be on by default. Basically, we
> > think
> > > > > that client-side auto-creation is actually a lot better-- hence
> this
> > > KIP
> > > > > :)
> > > > >
> > > > > But there is
> > > > > a functionality which allowed creation of topics if they don't
> exist
> > in
> > > > >
> > > > > the
> > > > >
> > > > > cluster and this behavior could be controlled by a config on the
> > server
> > > > > side. Now with this KIP we are allowing producer to make that
> > decision
> > > > > without any gateway on the server via configs. Any user who is not
> > > aware
> > > > >
> > > > > of
> > > > >
> > > > > such changes
> > > > > can accidentally create these topics and we are essentially
> removing
> > a
> > > > > config that exists in brokers today to block this accidental
> creation
> > > and
> > > > > allowing clients to take control.
> > > > >
> > > > > Again, I hope I'm not misinterpreting, but I don't see how this can
> > be
> > > > > turned on accidentally. The user would have to specifically turn
> this
> > > on
> > > > in
> > > > > the producer by setting the configuration key.
> > > > >
> > > > > I still didn't get any positives of not having server side configs?
> > if
> > > > you
> > > > > want to turn them on and allow any client to create topics set the
> > > > default
> > > > > to true
> > > > > and allow users who doesn't want to have this feature let them turn
> > > them
> > > > > off. It will be the exact behavior as it is today , as far as
> > producer
> > > is
> > > > > concerned. I am not
> > > > > understanding why not having server side configs to gateways such a
> > > hard
> > > > > requirement and this behavior exists today. As far I am concerned
> > this
> > > > > breaks the backward compatibility.
> > > > >
> > > > > The downside is that if we wanted to check a server side
> > configuration
> > > > > before sending the create topics request, the code would be more
> > > complex.
> > > > > The behavior would also not be consistent with how topic
> > auto-creation
> > > is
> > > > > handled in Kafka Streams.
> > > > >
> > > > > In general, it would be nice if we could keep the security and
> access
> > > > > control model simple and not introduce a lot of special cases and
> > > > > exceptions. Kafka has basically converged on using ACLs and
> > > > > CreateTopicPolicy classes to control who can create topics and
> where.
> > > > > Adding more knobs seems like a step backwards, especially when the
> > > > proposed
> > > > > knobs don't work consistently across components, and don't provide
> > true
> > > > > security.
> > > > >
> > > > > Maybe we should support an insecure mode where there are still
> users
> > > and
> > > > > ACLs. That would help people who don't want to set up Kerberos,
> etc.
> > > but
> > > > > still want to protect against misconfigured clients or accidents.
> > > Hadoop
> > > > > has something like this, and I think it was useful. NFS also
> > supported
> > > > > (supports?) a mode where you just pass whatever user ID you want
> and
> > > the
> > > > > system believes you. These things clearly don't protect against
> > > malicious
> > > > > users, but they can help set up policies when needed.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache.
> org (
> > > > > cmccabe@apache.org ) > wrote:
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > Thanks for taking a look.
> > > > >
> > > > > I'm not sure I follow the discussion about AdminClient.
> > > > >
> > > > > KafkaAdminClient
> > > > >
> > > > > has been around for about 2 years at this point as a public class.
> > > > >
> > > > > There
> > > > >
> > > > > are many programs that use it to automatically create topics. Kafka
> > > > > Streams does this, for example. If any of your users run Kafka
> > > > >
> > > > > Streams,
> > > > >
> > > > > they will be auto-creating topics, regardless of what setting you
> use
> > > > >
> > > > > for
> > > > >
> > > > > auto.create.topics.enable.
> > > > >
> > > > > A big part of the annoyance of auto-topic creation right now is
> that
> > > > >
> > > > > it is
> > > > >
> > > > > on by default. The new configuration proposed by KIP-487 wouldn't
> be.
> > > > > Users would have to explicitly opt in to the new behavior of
> > > > >
> > > > > client-side
> > > > >
> > > > > topic creation. If you run without security, you're already
> putting a
> > > > >
> > > > > huge
> > > > >
> > > > > amount of trust in your users. For example, you trust them not to
> > > > >
> > > > > delete
> > > > >
> > > > > records with the kafka-delete-records. sh (
> > > http://kafka-delete-records.
> > > > > sh/
> > > > > ) command, or delete topics with kafka-topics. sh (
> > > http://kafka-topics.
> > > > > sh/
> > > > > ). Trusting them not to set a certain config value seems minor in
> > > > > comparison, right?
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > > > >
> > > > > Hi,
> > > > > Even with policies one needs to implement that, so for every
> > > > >
> > > > > user who
> > > > >
> > > > > doesn't want a producer to create topics or have limits around
> > > > >
> > > > > partitions &
> > > > >
> > > > > replication factor they have to implement a policy. The KIP is
> > changing
> > > > > the behavior , it might not be introducing
> > > > >
> > > > > the
> > > > >
> > > > > new functionality but it will enable producers to override the
> create
> > > > >
> > > > > topic
> > > > >
> > > > > config settings on the broker. What I am asking for to provide a
> > > > >
> > > > > config
> > > > >
> > > > > that will disable auto creation of topics and if its enabled set
> some
> > > > >
> > > > > sane
> > > > >
> > > > > defaults so that clients can create a topic with in those limits. I
> > > > >
> > > > > don't
> > > > >
> > > > > see how this not related to this KIP.
> > > > > If the server config options as I suggested doesn't interest
> > > > >
> > > > > you at
> > > > >
> > > > > least have a default CreateTopicPolices in place.
> > > > > To give an example, In our environment we disable the
> > > > > auto.create.topic.enable and force users to go through a
> centralized
> > > > > service as we want capture more details about the topic creation
> and
> > > > > requirements. With this KIP, a producer can create a topic with no
> > > > >
> > > > > bounds.
> > > > >
> > > > > Another example max.message.size we define that at cluster level
> > > > >
> > > > > and one
> > > > >
> > > > > can override max.messsage.size at topic level, any misconfiguration
> > > > >
> > > > > at
> > > > >
> > > > > this
> > > > >
> > > > > will cause service degradation. Its not always about the rogue
> > > > >
> > > > > clients,
> > > > >
> > > > > users can easily misconfigure and can cause an outage. Again we can
> > > talk
> > > > > about CreateTopicPolicy but without having a
> > > > >
> > > > > default
> > > > >
> > > > > implementation and asking everyone to implement their own while
> > > > >
> > > > > changing
> > > > >
> > > > > the behavior in producer doesn't make sense to me.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me.
> > uk (
> > > > > ismael@juma.me.uk ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > I mentioned policies and the authorizer. For example, with
> > > > > CreateTopicPolicy, you can implement the limits you describe. If
> > > > >
> > > > > you
> > > > >
> > > > > have
> > > > >
> > > > > ideas of how that should be improved, please submit a KIP. My
> > > > >
> > > > > point is
> > > > >
> > > > > that
> > > > >
> > > > > this KIP is not introducing any new functionality with regards to
> > > > >
> > > > > what
> > > > >
> > > > > rogue clients can do. It's using the existing protocol that is
> > > > >
> > > > > already
> > > > >
> > > > > exposed via the AdminClient. So, I don't think we need to address
> > > > >
> > > > > it in
> > > > >
> > > > > this KIP. Does that make sense?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> > > > >
> > > > > kafka@ harsha. io ( kafka@harsha.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Ismael,
> > > > > Sure AdminClient can do that and we should've shipped a config or
> > > > >
> > > > > use
> > > > >
> > > > > the
> > > > >
> > > > > existing one to block that. Not all users are yet to upgrade to
> > > > >
> > > > > AdminClient
> > > > >
> > > > > and start using that to cause issues yet. In shared environment we
> > > > >
> > > > > should
> > > > >
> > > > > allow server to set sane defaults and not allow every client to go
> > > > >
> > > > > ahead
> > > > >
> > > > > create random no.of topic/partitions and replication factor. Even
> > > > >
> > > > > if
> > > > >
> > > > > the
> > > > >
> > > > > users want to allow topic creation proposed in the KIP , it makes
> > > > >
> > > > > sense to
> > > > >
> > > > > have some guards against the no.of partitions and replication
> > > > >
> > > > > factor.
> > > > >
> > > > > Authorizer is not always an answer to block requests and having
> > > > >
> > > > > users
> > > > >
> > > > > set
> > > > >
> > > > > server side configs to protect a multi-tenant environment is
> > > > >
> > > > > required.
> > > > >
> > > > > In a
> > > > >
> > > > > non-secure environment Authorizer is a blunt instrument either you
> > > > >
> > > > > end
> > > > >
> > > > > up
> > > > >
> > > > > blocking everyone or allowing everyone.
> > > > > I am asking to have server side that allow clients to create
> > > > >
> > > > > topics or
> > > > >
> > > > > not
> > > > >
> > > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > > replication-factor.
> > > > >
> > > > > -Harsha
> > > > >
> > > > > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com (
> > > ismaelj@gmail.com
> > > > > )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Harsha,
> > > > >
> > > > > Rogue clients can use the admin client to create topics and
> > > > >
> > > > > partitions.
> > > > >
> > > > > ACLs and policies can help in that case as well as this one.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha.
> > io
> > > (
> > > > > kafka@harsha.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > > Thanks for the KIP.
> > > > > "When server-side auto-creation is disabled, client-side
> > > > >
> > > > > auto-creation
> > > > >
> > > > > will try to use client-side configurations"
> > > > > If I understand correctly, this KIP is removing any server-side
> > > > >
> > > > > blocking
> > > > >
> > > > > client auto creation of topic?
> > > > > if so this will present potential issue of rogue client creating
> > > > >
> > > > > ton of
> > > > >
> > > > > topic-partitions and potentially bringing down the service for
> > > > >
> > > > > everyone
> > > > >
> > > > > or
> > > > >
> > > > > degrade the service itself.
> > > > > By reading the KIP its not clear to me that there is a clear way to
> > > > >
> > > > > block
> > > > >
> > > > > auto creation topics of all together from clients by server side
> > > > >
> > > > > config.
> > > > >
> > > > > Server side configs of default topic, partitions should take higher
> > > > > precedence and client shouldn't be able to create a topic with
> > > > >
> > > > > higher
> > > > >
> > > > > no.of
> > > > >
> > > > > partitions, replication than what server config specifies.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi all,
> > > > > I made some changes to the KIP. Hopefully this configuration change
> > > > >
> > > > > will
> > > > >
> > > > > make things a little clearer.
> > > > >
> > > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > > > https://cwiki.
> > > > > apache.org/confluence/display/KAFKA/ )
> > > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > > >
> > > > > Please let me know if you have any feedback or questions!
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache.
> > org (
> > > > > cmccabe@apache.org ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I think you bring up a good point. It would be better if we didn't
> > > > >
> > > > > ever
> > > > >
> > > > > have to set up client-side configuration for this feature, and
> > > > >
> > > > > KIP-464
> > > > >
> > > > > would let us skip this entirely.
> > > > >
> > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > > >
> > > > > Hi Mickael,
> > > > > I agree that KIP-464 works on newer brokers, but I was a bit
> > > > >
> > > > > worried
> > > > >
> > > > > how
> > > > >
> > > > > things would play out on older brokers that* do not *have KIP 464
> > > > >
> > > > > included.
> > > > >
> > > > > Is it enough to throw an error in this case when producer configs
> > > > >
> > > > > are
> > > > >
> > > > > not
> > > > >
> > > > > specified?
> > > > >
> > > > > I think the right thing to do would be to log an error message in
> > > > >
> > > > > the
> > > > >
> > > > > client. We will need to have that capability in any case, to cover
> > > > > scenarios like the client trying to auto-create a topic that they
> > > > >
> > > > > don't
> > > > >
> > > > > have permission to create. Or a client trying to create a topic on
> > > > >
> > > > > a
> > > > >
> > > > > broker
> > > > >
> > > > > so old that CreateTopicsRequest is not supported.
> > > > >
> > > > > The big downside to relying on KIP-464 is that it is a very recent
> > > > >
> > > > > feature
> > > > >
> > > > > -- so recent that it hasn't even made its way to any official
> > > > >
> > > > > Apache
> > > > >
> > > > > release. It's scheduled for the upcoming 2.4 release in a few
> > > > >
> > > > > months.
> > > > >
> > > > > So if you view this KIP as a step towards removing broker-side
> > > > > auto-create, you might want to support older brokers just to
> > > > >
> > > > > accelerate
> > > > >
> > > > > adoption, and hasten the day when we can finally flip broker-side
> > > > > auto-create to off (or even remove it entirely).
> > > > >
> > > > > I have to agree, though, that having client-side configurations for
> > > > >
> > > > > number
> > > > >
> > > > > of partitions and replication factor is messy. Maybe it would be
> > > > >
> > > > > worth
> > > > >
> > > > > it
> > > > >
> > > > > to restrict support to post-KIP-464 brokers, if we could avoid
> > > > >
> > > > > creating
> > > > >
> > > > > more configs.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> > > > >
> > > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > We can rely on KIP-464 which allows to omit the partition count or
> > > > > replication factor when creating a topic. In that case, the broker
> > > > >
> > > > > defaults
> > > > >
> > > > > are used.
> > > > >
> > > > > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Michael,
> > > > > That makes sense to me!
> > > > > To clarify, in the current state of the KIP, the producer does not
> > > > >
> > > > > rely
> > > > >
> > > > > on
> > > > >
> > > > > the broker to autocreate--if the broker's config is disabled, then
> > > > >
> > > > > the
> > > > >
> > > > > producer can autocreate on its own with a create topic request (the
> > > > >
> > > > > same
> > > > >
> > > > > type of request the admin client uses).
> > > > > However, if both configs are enabled, the broker will autocreate
> > > > >
> > > > > through
> > > > >
> > > > > a
> > > > >
> > > > > metadata request before the producer gets a chance. Of course, the
> > > > >
> > > > > way
> > > > >
> > > > > to
> > > > >
> > > > > avoid this, is to do as you suggested, and set
> > > > >
> > > > > the
> > > > >
> > > > > "allow_auto_topic_creation" field to false.
> > > > >
> > > > > I think the only thing we need to be careful with in this setup is
> > > > >
> > > > > without
> > > > >
> > > > > KIP 464, we can not use broker defaults for this topic. A user
> > > > >
> > > > > needs
> > > > >
> > > > > to
> > > > >
> > > > > specify the number of partition and replication factor in the
> > > > >
> > > > > config.
> > > > >
> > > > > An
> > > > >
> > > > > alternative to this is to have coded defaults for when these
> > > > >
> > > > > configs
> > > > >
> > > > > are
> > > > >
> > > > > unspecified, but it is not immediately apparent what these defaults
> > > > >
> > > > > should
> > > > >
> > > > > be.
> > > > >
> > > > > Thanks again for reading my KIP,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> > > > >
> > > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the response!
> > > > > In my opinion, it would be better if the producer did not rely at
> > > > >
> > > > > all
> > > > >
> > > > > on the broker auto create feature as this is what we're aiming to
> > > > > deprecate. When requesting metadata we can set the
> > > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > > creation. Then if the topic is not existing, send a
> > > > >
> > > > > CreateTopicRequest.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Currently the way it is implemented, the broker auto-creation
> > > > >
> > > > > configuration
> > > > >
> > > > > takes precedence. The producer will not use the CreateTopics
> > > > >
> > > > > request.
> > > > >
> > > > > (Technically it can--but the topic will already be created
> > > > >
> > > > > through
> > > > >
> > > > > the
> > > > >
> > > > > broker, so it will never try to create the topic.) It is possible
> > > > >
> > > > > to
> > > > >
> > > > > change this however, and I'd be happy to
> > > > >
> > > > > discuss
> > > > >
> > > > > the
> > > > >
> > > > > benefits of this alternative.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> > > > >
> > > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP!
> > > > >
> > > > > In case auto creation is enabled on both the client and server,
> > > > >
> > > > > will
> > > > >
> > > > > the producer still use the AdminClient (CreateTopics request)
> > > > >
> > > > > to
> > > > >
> > > > > create topics? and not rely on the broker auto create. I'm
> > > > >
> > > > > guessing the
> > > > >
> > > > > answer is yes but can you make it explicit.
> > > > >
> > > > > Thank you
> > > > >
> > > > > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi,
> > > > > Just a friendly reminder to take a look at this KIP if you
> > > > >
> > > > > have
> > > > >
> > > > > the
> > > > >
> > > > > time.
> > > > >
> > > > > I was thinking about broker vs. client default precedence,
> > > > >
> > > > > and I
> > > > >
> > > > > think it
> > > > >
> > > > > makes sense to keep the broker as the default used when both
> > > > >
> > > > > client-side
> > > > >
> > > > > and broker-side defaults are configured. The idea is that
> > > > >
> > > > > there
> > > > >
> > > > > would be
> > > > >
> > > > > pretty clear documentation, and that many systems with
> > > > >
> > > > > configurations
> > > > >
> > > > > that
> > > > >
> > > > > the client could not change would likely have the auto-create
> > > > >
> > > > > default
> > > > >
> > > > > off.
> > > > >
> > > > > (In cloud for example).
> > > > >
> > > > > It also seems like in most cases, the consumer config
> > > > > ' allow. auto. create. topics ( http://allow.auto.create.topics/
> ) '
> > > was
> > > > > created to actually prevent
> > > > >
> > > > > the
> > > > >
> > > > > creation
> > > > >
> > > > > of
> > > > >
> > > > > topics, so the loss of creation functionality will not be a
> > > > >
> > > > > big
> > > > >
> > > > > problem.
> > > > >
> > > > > I'm happy to discuss any other compatibility problems or
> > > > >
> > > > > components
> > > > >
> > > > > of
> > > > >
> > > > > this KIP.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > I was looking at this KIP again, and there is a decision I
> > > > >
> > > > > made
> > > > >
> > > > > that I
> > > > >
> > > > > think is worth discussing.
> > > > >
> > > > > In the case where both the broker and producer's
> > > > > 'auto.create.topics.enable' are set to true, we have to
> > > > >
> > > > > choose
> > > > >
> > > > > either
> > > > >
> > > > > the
> > > > >
> > > > > broker configs or the producer configs for the replication
> > > > > factor/partitions.
> > > > >
> > > > > Currently, the decision is to have the broker defaults take
> > > > >
> > > > > precedence.
> > > > >
> > > > > (It is easier to do this in the implementation.) It also
> > > > >
> > > > > makes
> > > > >
> > > > > some
> > > > >
> > > > > sense
> > > > >
> > > > > for this behavior to take precedence since this behavior
> > > > >
> > > > > already
> > > > >
> > > > > occurs as
> > > > >
> > > > > the default.
> > > > >
> > > > > However, I was wondering if it would be odd for those who
> > > > >
> > > > > can
> > > > >
> > > > > only
> > > > >
> > > > > see
> > > > >
> > > > > the
> > > > >
> > > > > producer side to set configs for replication
> > > > >
> > > > > factor/partitions
> > > > >
> > > > > and
> > > > >
> > > > > see
> > > > >
> > > > > different results. Currently the documentation for the
> > > > >
> > > > > config
> > > > >
> > > > > states
> > > > >
> > > > > that
> > > > >
> > > > > the config values are only used when the broker config is
> > > > >
> > > > > not
> > > > >
> > > > > enabled,
> > > > >
> > > > > but
> > > > >
> > > > > this might not always be clear to the user. Changing the
> > > > >
> > > > > code
> > > > >
> > > > > to
> > > > >
> > > > > have
> > > > >
> > > > > the
> > > > >
> > > > > producer's configurations take precedence is possible, but
> > > > >
> > > > > I
> > > > >
> > > > > was
> > > > >
> > > > > wondering
> > > > >
> > > > > what everyone thought.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Just a quick update--
> > > > >
> > > > > It seems that enabling both the broker and producer
> > > > >
> > > > > configs
> > > > >
> > > > > works
> > > > >
> > > > > fine,
> > > > >
> > > > > except that the broker configurations for partitions,
> > > > >
> > > > > replication
> > > > >
> > > > > factor
> > > > >
> > > > > take precedence.
> > > > > I don't know if that is something we would want to
> > > > >
> > > > > change, but
> > > > >
> > > > > I'll be
> > > > >
> > > > > updating the KIP for now to reflect this. Perhaps we would
> > > > >
> > > > > want to
> > > > >
> > > > > add more
> > > > >
> > > > > to the documentation of the the producer configs to
> > > > >
> > > > > clarify.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Colin,
> > > > >
> > > > > Thanks for looking at the KIP. I can definitely add to
> > > > >
> > > > > the
> > > > >
> > > > > title
> > > > >
> > > > > to
> > > > >
> > > > > make
> > > > >
> > > > > it more clear.
> > > > >
> > > > > It makes sense that both configurations could be turned
> > > > >
> > > > > on
> > > > >
> > > > > since
> > > > >
> > > > > there
> > > > >
> > > > > are many cases where the user can not control the
> > > > >
> > > > > server-side
> > > > >
> > > > > configurations. I was a little concerned about how both
> > > > >
> > > > > interacting
> > > > >
> > > > > would
> > > > >
> > > > > work out -- if there would be to many requests for new
> > > > >
> > > > > topics,
> > > > >
> > > > > for
> > > > >
> > > > > example.
> > > > >
> > > > > But it since it does make sense to allow both
> > > > >
> > > > > configurations
> > > > >
> > > > > enabled, I
> > > > >
> > > > > will test out some scenarios and I'll change the KIP to
> > > > >
> > > > > support
> > > > >
> > > > > this.
> > > > >
> > > > > I also agree with documentation about distinguishing the
> > > > >
> > > > > differences. I
> > > > >
> > > > > was having some trouble with the wording but I like the
> > > > >
> > > > > phrases
> > > > >
> > > > > "server-side" and "client-side." That's a good
> > > > >
> > > > > distinction I
> > > > >
> > > > > can
> > > > >
> > > > > use
> > > > >
> > > > > when
> > > > >
> > > > > describing.
> > > > >
> > > > > I'll try to update the KIP soon keeping everyone's input
> > > > >
> > > > > in
> > > > >
> > > > > mind.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> > > > >
> > > > > cmccabe@ apache. org ( cmccabe@apache.org )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP. This seems like a good step towards
> > > > >
> > > > > removing
> > > > >
> > > > > server-side topic auto-creation.
> > > > >
> > > > > We should add included "client-side" to the title of
> > > > >
> > > > > the KIP
> > > > >
> > > > > somewhere,
> > > > >
> > > > > to make it clear that we're talking about client-side
> > > > >
> > > > > auto
> > > > >
> > > > > creation.
> > > > >
> > > > > The KIP says:
> > > > >
> > > > > In order to automatically create topics with the
> > > > >
> > > > > producer, the
> > > > >
> > > > > producer's
> > > > >
> > > > > auto.create.topics.enable config must be set to true
> > > > >
> > > > > and
> > > > >
> > > > > the
> > > > >
> > > > > broker
> > > > >
> > > > > config should be set to false
> > > > >
> > > > > From a user's point of view, this seems
> > > > >
> > > > > counter-intuitive.
> > > > >
> > > > > In
> > > > >
> > > > > order to
> > > > >
> > > > > auto-create topics the broker's
> > > > >
> > > > > auto.create.topics.enable
> > > > >
> > > > > config
> > > > >
> > > > > should be
> > > > >
> > > > > set to false? It seems like the server-side
> > > > >
> > > > > auto-create is
> > > > >
> > > > > unrelated to
> > > > >
> > > > > the client-side auto-create. We could have both turned
> > > > >
> > > > > on
> > > > >
> > > > > (and
> > > > >
> > > > > I'm
> > > > >
> > > > > sure
> > > > >
> > > > > that in the real world, people will try this
> > > > >
> > > > > configuration...)
> > > > >
> > > > > There's no
> > > > >
> > > > > reason not to support this, I think.
> > > > >
> > > > > We should add some documentation explaining the
> > > > >
> > > > > difference
> > > > >
> > > > > between
> > > > >
> > > > > server-side and client-side auto-creation. Without
> > > > >
> > > > > documentation,
> > > > >
> > > > > an admin
> > > > >
> > > > > might think that they had disabled all forms of
> > > > >
> > > > > auto-creation by
> > > > >
> > > > > setting
> > > > >
> > > > > the -side setting to false-- but this is not the case,
> > > > >
> > > > > of
> > > > >
> > > > > course.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > > >
> > > > > Hi Dhruvil,
> > > > >
> > > > > Thanks for reading the KIP!
> > > > > That was the general idea for deprecation. We would
> > > > >
> > > > > log a
> > > > >
> > > > > warning
> > > > >
> > > > > when the
> > > > >
> > > > > config is enabled on the broker.
> > > > > I also believe that there would be a change to
> > > > >
> > > > > documentation.
> > > > >
> > > > > If there is anything else that should be done, please
> > > > >
> > > > > let
> > > > >
> > > > > me
> > > > >
> > > > > know!
> > > > >
> > > > > Justine
> > > > >
> > > > > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> > > > >
> > > > > dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP, this is great!
> > > > >
> > > > > Could you add some more information about what
> > > > >
> > > > > deprecating
> > > > >
> > > > > the
> > > > >
> > > > > broker
> > > > >
> > > > > configuration means? Would we log a warning in the
> > > > >
> > > > > logs
> > > > >
> > > > > when
> > > > >
> > > > > auto
> > > > >
> > > > > topic
> > > > >
> > > > > creation is enabled on the broker, for example?
> > > > >
> > > > > Thanks,
> > > > > Dhruvil
> > > > >
> > > > > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > I'd like to start a discussion thread for KIP-487. This KIP plans
> > > > >
> > > > > to
> > > > >
> > > > > deprecate the current system of
> > > > >
> > > > > auto-creating
> > > > >
> > > > > topics
> > > > >
> > > > > through requests to the metadata and give the
> > > > >
> > > > > producer the
> > > > >
> > > > > ability to
> > > > >
> > > > > automatically create topics instead.
> > > > >
> > > > > More information can be found here:
> > > > >
> > > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > > > https://cwiki.
> > > > > apache.org/confluence/display/KAFKA/ )
> > > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > > >
> > > > > Thank you,
> > > > > Justine Olshan
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Colin McCabe <cm...@apache.org>.

On Thu, Aug 8, 2019, at 11:51, Jun Rao wrote:
> Hi, Justine,
> 
> Thanks for the KIP. Overall, it seems to be a good improvement.
> 
> However, I think Harsha's point seems reasonable. We had
> auto.create.topics.enable config on the broker to allow admins to disable
> topic creation from the producer/consumer clients before we had the
> security feature. The need for that config is reduced with the security
> feature, but may still be present since not all places have security
> enabled. It's true that a non-secured environment is vulnerable to some
> additional attacks, but producer/consumer are the most common way for a
> user to interact with the broker. So, keeping that config for backward
> compatibility could still be useful if it's not introducing too much effort
> or extra confusion.
> 
> Here is a one potential alternative way that I was thinking. We add a new
> field in the CreateTopicRequest to indicate whether it's from the producer
> or not. If auto.create.topics.enable is false, CreateTopicRequest from the
> producer will be rejected.

Hi Jun,

Thanks for taking a look.

One major goal of this KIP is to allow sysadmins to disable server-side topic auto-creation through the metadata request, while still enabling client-side auto-creation for producers.  So perhaps we could have three values for "auto.create.topics.enable" on the broker: true, false, and client-side.  "client-side" would indicate that auto topic creation through the metadata request would be blocked, but through create topics request would be enabled.

> We probably don't need to introduce the new
> config (which seems a bit hard to explain) in the producer. Instead, the
> new producer always uses MetadataRequest with AllowAutoTopicCreation set to
> false to get the metadata and if the metadata is not present, send the
> new CreateTopicRequest
> (assuming the broker supports it) to try to create the topic automatically.
> Whether the creation is allowed or not will be determined by the broker.
> This will make the behavior backward compatible and we can still achieve
> the main goal of the KIP, which is not relying on MetadataRequest for topic
> creation. What do you think?

I agree that it may not be necessary to introduce a new configuration for the producer, if we add the server-side access control you suggested.  However, new clients will still need to support auto-creation on older brokers, so that producers that rely on it keep working across a client upgrade.  That means attempting to do it through MetadataRequest, if the new mechanism is not supported.

From a compatibility point of view, if we rely on a new field in CreateTopicsRequest, we definitely won't be able to use the CreateTopicsRequest mechanism against any broker but a 2.4+ one.  That's probably fine (and consistent with some of the other discussion we've been having here) but it worth noting.

best,
Colin


> 
> Thanks,
> 
> Jun
> 
> On Thu, Aug 8, 2019 at 1:34 AM M. Manna <ma...@gmail.com> wrote:
> 
> > Hi,
> >
> > If I may, perhaps you could simplify everything by using only
> > 'auto.create.topics.enable' as a value along with true. In other words, the
> > public interfaces section should only have [true,auto.create.topics.enable,
> > false].
> >
> > The reason for this is that auto.create.topics.enable is already known to
> > users as a "Server-SIde" config. So all you are saying is
> >
> > a) To avoid day 1 impact, it will follow whatever auto.create.topics.enable
> > value is set.
> > b) False means - no client side topic creation
> > c) True means client side topic creation.
> >
> > It saves creating 2 more new strings :). But not too expensive anyway.
> >
> > Also, when you deprecate auto.create.topics.enable - you must provide
> > sufficient logic to ensure that things like rolling upgrade doesn't
> > temporarily break anything. I apologise if you have already accounted for
> > this, but wanted to mention since I didn't notice this on the KIP.
> >
> > Let me know how this sounds.
> >
> > Regards,
> >
> > On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jo...@confluent.io> wrote:
> >
> > > Hi Harsha,
> > >
> > > I think my message may have gotten lost in all the others.
> > >
> > > Two of the goals of this KIP are to 1) allow auto-creation on specific
> > > clients when the broker default is false and 2) eventually replace the
> > > broker config.
> > >
> > > In order to accomplish these two goals, we need the producer to be able
> > to
> > > create topics despite the broker config. (How can we replace a function
> > > when we rely on it?)
> > > I think at this point we have a fundamental disagreement in what we
> > should
> > > allow the producer to do.
> > > In my previous message I mentioned a config that would allow for the
> > broker
> > > to prevent producer auto-creation. (It would be disabled by default.) It
> > > would fix your issue for now, but could lead to more complications later.
> > >
> > > Thank you,
> > > Justine
> > >
> > >
> > > On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz>
> > wrote:
> > > >
> > > > > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> > > > >
> > > > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmccabe@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> > > > >
> > > > > Hi Colin,
> > > > > "Hmm... I'm not sure I follow. Users don't have to build their own
> > > > > tooling, right? They can use any of the shell scripts that we've
> > > shipped
> > > > in
> > > > > the last few releases. For example, if any of your users run it, this
> > > > shell
> > > > > script will delete all of the topics from your non-security-enabled
> > > > > cluster:
> > > > >
> > > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > > --bootstrap-server localhost:9092 --delete
> > > > > --topic
> > > > >
> > > > > They will need to fill in the correct bootstrap servers list, of
> > > course,
> > > > > not localhost. This deletion script will work on some pretty old
> > > brokers,
> > > > > even back to the 0.10 releases. It seems a little odd to trust your
> > > users
> > > > > with this power, but not trust them to avoid changing a particular
> > > > > configuration key."
> > > > >
> > > > > The above will blocked by the server if we set delete.topic.enable to
> > > > > false and thats exactly what I am asking for.
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > I was wondering if someone was going to bring up that configuration
> > :)
> > > > >
> > > > > it's an interesting complication, but globally disabling topic
> > deletion
> > > > is
> > > > > not very practical for most use-cases.
> > > > >
> > > > > In any case, there are plenty of other bad things that users with
> > full
> > > > > permissions can do that aren't blocked by any server configuration.
> > For
> > > > > example, they can delete every record in every topic. I can write a
> > > > script
> > > > > for that too, and there's no server configuration you can set to
> > > disable
> > > > > it. Or I could simply create hundreds of thousands of topics, until
> > > > cluster
> > > > > performance becomes unacceptable (this will be even more of a problem
> > > if
> > > > > someone configured delete.topic.enable as false). Or publish bad data
> > > to
> > > > > every topic, etc. etc.
> > > > >
> > > > > The point I'm trying to make here is that you can't rely on these
> > kind
> > > of
> > > > > server-side configurations for security. At most, they're a way to
> > set
> > > up
> > > > > certain very simple policies. But the policies are so simple that
> > > they're
> > > > > hardly ever useful any more.
> > > > >
> > > > > For example, if the problem you want to solve is that you want a user
> > > to
> > > > > only be able to create 50 topics and not delete anyone else's topics,
> > > you
> > > > > can solve that with a CreateTopicsPolicy that limits the number of
> > > > topics,
> > > > > and some ACLs. There's no combination of auto.create.topics.enable
> > and
> > > > > delete.topic.enable that will help here.
> > > > >
> > > > > Hi Colin,
> > > > >
> > > > >             Well you gave the example that a user can delete the
> > topics
> > > > > just by running that script  :).
> > > > >
> > > > > I understand there are open APIs in Kafka and can lead to rogue
> > clients
> > > > > taking advantage of it without proper security in place.
> > > > >
> > > > > What I am asking so far in this thread is , this KIP is changing the
> > > > > producer behavior and its not backward compatible.
> > > > >
> > > > > The change is backwards compatible. The default will still be
> > > server-side
> > > > > topic auto-creation, just like now.
> > > > >
> > > > You will have to specifically change the producer config to get the new
> > > > > behavior.
> > > > >
> > > >
> > > >
> > > > I disagree.  Today server can turn off the topic creation  neither
> > > producer
> > > > or consumer can create a topic. With this KIP , producer can create a
> > > topic
> > > > by turning on client side config when server side config is turned off.
> > > >
> > > >
> > > > We can still achieve
> > > > > the main goal of the KIP which is to change MetadataRequest  creating
> > > > > topics and send a CreateTopicRequest from Producer and also keep the
> > > > server
> > > > > side config to have precedence.  This KIP originally written to have
> > > > server
> > > > > side preference and there is not much explanation why it changed to
> > > have
> > > > > Producer side preference.
> > > > >
> > > > > Arguing that AdminClient can do that and so we are going to make
> > > Producer
> > > > > do the same doesn't make sense.
> > > > >
> > > > > "The downside is that if we wanted to check a server side
> > configuration
> > > > > before sending the create topics request, the code would be more
> > > complex.
> > > > > The behavior would also not be consistent with how topic
> > auto-creation
> > > is
> > > > > handled in Kafka Streams."
> > > > >
> > > > > I am not sure why you need to check server side configuration before
> > > > > sending create topics request. A user enables producer side config to
> > > > > create topics.
> > > > > Producer sends a request to the broker and if the broker has
> > > > > auto.topic.create.enable to true (default) it will allow creation of
> > > > > topics. If it set to false it returns error back to the client.
> > > > >
> > > > > auto.topic.create.enable has never affected CreateTopicsRequest. If
> > you
> > > > > submit a CreateTopicsRequest and you are authorized, a topic will be
> > > > > created, regardless of what the value of auto.topic.create.enable is.
> > > > This
> > > > > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?)
> > > > >
> > > > > I don't see how this behavior will be different in Kafka streams. By
> > > > > default server allows the topic creation and with this KIP, It will
> > > only
> > > > > allow creation of topic when both producer and server side are turned
> > > on.
> > > > > Its exactly the same behavior in KIP-361.
> > > > >
> > > > > I suggest running a streams job as a test. It creates the topics it
> > > needs
> > > > > using AdminClient. The value of auto.topic.create.enable is not
> > > relevant.
> > > > > Whether it is true or false, the topics will still be created.
> > > > >
> > > > > "In general, it would be nice if we could keep the security and
> > access
> > > > > control model simple and not introduce a lot of special cases and
> > > > > exceptions. Kafka has basically converged on using ACLs and
> > > > > CreateTopicPolicy classes to control who can create topics and where.
> > > > > Adding more knobs seems like a step backwards, especially when the
> > > > proposed
> > > > > knobs don't work consistently across components, and don't provide
> > true
> > > > > security." This is not about access control at all. Shipping sane
> > > > defaults
> > > > > should be prioritized.
> > > > >
> > > > > I don't think the default is really the question here. I think
> > everyone
> > > > > agrees that the default for client-side auto-topic creation should be
> > > > off.
> > > > >
> > > > > My point goes back to KIP-361 why we kept the server side config to
> > > have
> > > > > preference. We can keep bringing back the discussion of security
> > > policies
> > > > > but a producer client overiding server side setting is not just
> > > security
> > > > > policy issue .
> > > > >
> > > > > "Producer client can override server side setting and not consumer
> > and
> > > > > also producer can only override creation of topic but no the
> > > replication
> > > > > and partitions. " This doesn't make sense to me and why we want to
> > > > > introduce such a confusing behavior.
> > > > >
> > > > > The scenarios that you're describing (such as dealing with a poorly
> > > > > configured client that tries to create topics it should not) do seem
> > to
> > > > be
> > > > > about access control.
> > > > >
> > > > > We keep talking about CreateTopicPolicy and yet we don't have default
> > > one
> > > > > and asking every user of Kafka implement their own doesn't make sense
> > > > here.
> > > > >
> > > > > The point of CreateTopicPolicy is exactly that you can implement your
> > > > own,
> > > > > though. It's a way of customizing what the policy is in your specific
> > > > > cluster.
> > > > >
> > > > > I agree that most people don't want to write a custom policy. But
> > most
> > > of
> > > > > those people are satisfied with just setting up ACLs to set up simple
> > > > > policies like this user can create topics, that user can't, etc.
> > That's
> > > > why
> > > > > I suggested adding support for ACLs to insecure clusters might be
> > > > helpful.
> > > > >
> > > > > Also, just as a side note, wouldn't it would be impossible for us to
> > > > > specify a default CreateTopicsPolicy that did anything at this point
> > > > anyway
> > > > > without breaking backwards compatibility? Maybe I'm misunderstanding
> > > the
> > > > > proposal.
> > > > >
> > > > > I am asking about exact behavior that we shipped for consumer side
> > > > https:/
> > > > > / cwiki. apache. org/ confluence/ display/ KAFKA/
> > > > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > > > (
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > > > )
> > > > >
> > > > > I still didn't get any response why this behavior shouldn't be
> > exactly
> > > > > like Kafka consumer and why do we want to have producer to overrider
> > > > server
> > > > > side config and while not allowing consumer to do so. We are not even
> > > > > allowing the same contract and this will cause more confusion from
> > the
> > > > > users standpoint.
> > > > >
> > > > > Hmm. The consumer already has a way to override the server side
> > > > > configuration. See KIP-361: Add Consumer Configuration to Disable
> > Auto
> > > > > Topic Creation. This lets the consumer skip auto-creating topics,
> > even
> > > if
> > > > > auto-creation is enabled on the broker.
> > > > >
> > > > > Yes that's what I am saying and it doesn't allow topic creation if
> > the
> > > > > server side auto-creation is disabled and in consumer its enabled.
> >  I
> > > > > would like to see the exact behavior for producer as stated in
> > KIP-361.
> > > > >
> > > > > To be fair, the KIP should probably discuss why we don't want to
> > > > implement
> > > > > client-side topic creation in the consumer in "rejected
> > alternatives."
> > > > > Maybe Justine can add more context here in the KIP.
> > > > >
> > > > > The last time we talked about this, the reasoning was that we wanted
> > to
> > > > > eventually get rid of consumer-side auto-topic creation entirely, and
> > > so
> > > > it
> > > > > wasn't worth implementing an improved way of doing it. Producer side
> > > auto
> > > > > topic-creation, in contrast, will probably stick around, although
> > we'd
> > > > like
> > > > > to deprecate and remove the problematic server-side mechanism over
> > > time.
> > > > >
> > > > > We should implement the producer side topic creation with proper
> > create
> > > > > topic request  and I am in favor of this KIP.
> > > > >
> > > > > All I am asking is don't make the producer to override server side
> > > config
> > > > > and keep it similar to KIP-361 just like consumer, it honors what set
> > > on
> > > > > server side which takes the precedence.   Changing this behavior in
> > > > > producer is backward incompatible and will catch users by surprise.
> > > > >
> > > > > All arguments for enforcing producer side overriding config can still
> > > be
> > > > > achieved. Server side auto creation turned on by default and  any new
> > > > > producer client setting their auto creation config to true will
> > create
> > > > > topics in default behavior and any user who set server side to false
> > > and
> > > > > upgrades to latest kafka with this KIP will not see any unwanted
> > > behavior
> > > > > from clients.  IMO this is not a unreasonable request on this KIP and
> > > > this
> > > > > requested behavior is exactly what the KIP initially proposed.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Harsha
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache. org (
> > > > > cmccabe@apache.org ) > wrote:
> > > > >
> > > > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > > > >
> > > > > Not sure how the AdminClient applies here, It is an external API and
> > is
> > > > > not part of KafkaProducer so any user who updates to latest version
> > of
> > > > > Kafka with this feature get to create the topics.
> > > > > They have to build a tooling around AdminClient allow themselves to
> > > > >
> > > > > create
> > > > >
> > > > > topics.
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > Hmm... I'm not sure I follow. Users don't have to build their own
> > > > tooling,
> > > > > right? They can use any of the shell scripts that we've shipped in
> > the
> > > > last
> > > > > few releases. For example, if any of your users run it, this shell
> > > script
> > > > > will delete all of the topics from your non-security-enabled cluster:
> > > > >
> > > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > > --bootstrap-server localhost:9092 --delete
> > > > > --topic
> > > > >
> > > > > They will need to fill in the correct bootstrap servers list, of
> > > course,
> > > > > not localhost. This deletion script will work on some pretty old
> > > brokers,
> > > > > even back to the 0.10 releases. It seems a little odd to trust your
> > > users
> > > > > with this power, but not trust them to avoid changing a particular
> > > > > configuration key.
> > > > >
> > > > > There is no behavior in Kafka producer that allowed users to delete
> > the
> > > > > topics or delete the records. So citing them as an example doesn't
> > > makes
> > > > > sense in this context.
> > > > >
> > > > > I think Kafka Streams is relevant here. After all, it's software that
> > > we
> > > > > ship as part of the official Kafka release. And it auto-creates
> > topics
> > > > even
> > > > > when auto.create.topics.enable is set to false on the broker.
> > > > >
> > > > > I think that auto.create.topics.enable was never intended as a
> > security
> > > > > setting to control access. It was intended as a way to disable the
> > > > > broker-side auto-create feature specifically, because there were some
> > > > > problems with that specific feature. Broker-side auto-creation is
> > > > > frustrating because it's on by default, and because it can
> > auto-create
> > > > > topics even if the producer or consumer didn't explicitly ask for
> > that.
> > > > > Neither of those problems applies to this KIP: producers have to
> > > > > specifically opt in, and it won't be on by default. Basically, we
> > think
> > > > > that client-side auto-creation is actually a lot better-- hence this
> > > KIP
> > > > > :)
> > > > >
> > > > > But there is
> > > > > a functionality which allowed creation of topics if they don't exist
> > in
> > > > >
> > > > > the
> > > > >
> > > > > cluster and this behavior could be controlled by a config on the
> > server
> > > > > side. Now with this KIP we are allowing producer to make that
> > decision
> > > > > without any gateway on the server via configs. Any user who is not
> > > aware
> > > > >
> > > > > of
> > > > >
> > > > > such changes
> > > > > can accidentally create these topics and we are essentially removing
> > a
> > > > > config that exists in brokers today to block this accidental creation
> > > and
> > > > > allowing clients to take control.
> > > > >
> > > > > Again, I hope I'm not misinterpreting, but I don't see how this can
> > be
> > > > > turned on accidentally. The user would have to specifically turn this
> > > on
> > > > in
> > > > > the producer by setting the configuration key.
> > > > >
> > > > > I still didn't get any positives of not having server side configs?
> > if
> > > > you
> > > > > want to turn them on and allow any client to create topics set the
> > > > default
> > > > > to true
> > > > > and allow users who doesn't want to have this feature let them turn
> > > them
> > > > > off. It will be the exact behavior as it is today , as far as
> > producer
> > > is
> > > > > concerned. I am not
> > > > > understanding why not having server side configs to gateways such a
> > > hard
> > > > > requirement and this behavior exists today. As far I am concerned
> > this
> > > > > breaks the backward compatibility.
> > > > >
> > > > > The downside is that if we wanted to check a server side
> > configuration
> > > > > before sending the create topics request, the code would be more
> > > complex.
> > > > > The behavior would also not be consistent with how topic
> > auto-creation
> > > is
> > > > > handled in Kafka Streams.
> > > > >
> > > > > In general, it would be nice if we could keep the security and access
> > > > > control model simple and not introduce a lot of special cases and
> > > > > exceptions. Kafka has basically converged on using ACLs and
> > > > > CreateTopicPolicy classes to control who can create topics and where.
> > > > > Adding more knobs seems like a step backwards, especially when the
> > > > proposed
> > > > > knobs don't work consistently across components, and don't provide
> > true
> > > > > security.
> > > > >
> > > > > Maybe we should support an insecure mode where there are still users
> > > and
> > > > > ACLs. That would help people who don't want to set up Kerberos, etc.
> > > but
> > > > > still want to protect against misconfigured clients or accidents.
> > > Hadoop
> > > > > has something like this, and I think it was useful. NFS also
> > supported
> > > > > (supports?) a mode where you just pass whatever user ID you want and
> > > the
> > > > > system believes you. These things clearly don't protect against
> > > malicious
> > > > > users, but they can help set up policies when needed.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache. org (
> > > > > cmccabe@apache.org ) > wrote:
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > Thanks for taking a look.
> > > > >
> > > > > I'm not sure I follow the discussion about AdminClient.
> > > > >
> > > > > KafkaAdminClient
> > > > >
> > > > > has been around for about 2 years at this point as a public class.
> > > > >
> > > > > There
> > > > >
> > > > > are many programs that use it to automatically create topics. Kafka
> > > > > Streams does this, for example. If any of your users run Kafka
> > > > >
> > > > > Streams,
> > > > >
> > > > > they will be auto-creating topics, regardless of what setting you use
> > > > >
> > > > > for
> > > > >
> > > > > auto.create.topics.enable.
> > > > >
> > > > > A big part of the annoyance of auto-topic creation right now is that
> > > > >
> > > > > it is
> > > > >
> > > > > on by default. The new configuration proposed by KIP-487 wouldn't be.
> > > > > Users would have to explicitly opt in to the new behavior of
> > > > >
> > > > > client-side
> > > > >
> > > > > topic creation. If you run without security, you're already putting a
> > > > >
> > > > > huge
> > > > >
> > > > > amount of trust in your users. For example, you trust them not to
> > > > >
> > > > > delete
> > > > >
> > > > > records with the kafka-delete-records. sh (
> > > http://kafka-delete-records.
> > > > > sh/
> > > > > ) command, or delete topics with kafka-topics. sh (
> > > http://kafka-topics.
> > > > > sh/
> > > > > ). Trusting them not to set a certain config value seems minor in
> > > > > comparison, right?
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > > > >
> > > > > Hi,
> > > > > Even with policies one needs to implement that, so for every
> > > > >
> > > > > user who
> > > > >
> > > > > doesn't want a producer to create topics or have limits around
> > > > >
> > > > > partitions &
> > > > >
> > > > > replication factor they have to implement a policy. The KIP is
> > changing
> > > > > the behavior , it might not be introducing
> > > > >
> > > > > the
> > > > >
> > > > > new functionality but it will enable producers to override the create
> > > > >
> > > > > topic
> > > > >
> > > > > config settings on the broker. What I am asking for to provide a
> > > > >
> > > > > config
> > > > >
> > > > > that will disable auto creation of topics and if its enabled set some
> > > > >
> > > > > sane
> > > > >
> > > > > defaults so that clients can create a topic with in those limits. I
> > > > >
> > > > > don't
> > > > >
> > > > > see how this not related to this KIP.
> > > > > If the server config options as I suggested doesn't interest
> > > > >
> > > > > you at
> > > > >
> > > > > least have a default CreateTopicPolices in place.
> > > > > To give an example, In our environment we disable the
> > > > > auto.create.topic.enable and force users to go through a centralized
> > > > > service as we want capture more details about the topic creation and
> > > > > requirements. With this KIP, a producer can create a topic with no
> > > > >
> > > > > bounds.
> > > > >
> > > > > Another example max.message.size we define that at cluster level
> > > > >
> > > > > and one
> > > > >
> > > > > can override max.messsage.size at topic level, any misconfiguration
> > > > >
> > > > > at
> > > > >
> > > > > this
> > > > >
> > > > > will cause service degradation. Its not always about the rogue
> > > > >
> > > > > clients,
> > > > >
> > > > > users can easily misconfigure and can cause an outage. Again we can
> > > talk
> > > > > about CreateTopicPolicy but without having a
> > > > >
> > > > > default
> > > > >
> > > > > implementation and asking everyone to implement their own while
> > > > >
> > > > > changing
> > > > >
> > > > > the behavior in producer doesn't make sense to me.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me.
> > uk (
> > > > > ismael@juma.me.uk ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > I mentioned policies and the authorizer. For example, with
> > > > > CreateTopicPolicy, you can implement the limits you describe. If
> > > > >
> > > > > you
> > > > >
> > > > > have
> > > > >
> > > > > ideas of how that should be improved, please submit a KIP. My
> > > > >
> > > > > point is
> > > > >
> > > > > that
> > > > >
> > > > > this KIP is not introducing any new functionality with regards to
> > > > >
> > > > > what
> > > > >
> > > > > rogue clients can do. It's using the existing protocol that is
> > > > >
> > > > > already
> > > > >
> > > > > exposed via the AdminClient. So, I don't think we need to address
> > > > >
> > > > > it in
> > > > >
> > > > > this KIP. Does that make sense?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> > > > >
> > > > > kafka@ harsha. io ( kafka@harsha.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Ismael,
> > > > > Sure AdminClient can do that and we should've shipped a config or
> > > > >
> > > > > use
> > > > >
> > > > > the
> > > > >
> > > > > existing one to block that. Not all users are yet to upgrade to
> > > > >
> > > > > AdminClient
> > > > >
> > > > > and start using that to cause issues yet. In shared environment we
> > > > >
> > > > > should
> > > > >
> > > > > allow server to set sane defaults and not allow every client to go
> > > > >
> > > > > ahead
> > > > >
> > > > > create random no.of topic/partitions and replication factor. Even
> > > > >
> > > > > if
> > > > >
> > > > > the
> > > > >
> > > > > users want to allow topic creation proposed in the KIP , it makes
> > > > >
> > > > > sense to
> > > > >
> > > > > have some guards against the no.of partitions and replication
> > > > >
> > > > > factor.
> > > > >
> > > > > Authorizer is not always an answer to block requests and having
> > > > >
> > > > > users
> > > > >
> > > > > set
> > > > >
> > > > > server side configs to protect a multi-tenant environment is
> > > > >
> > > > > required.
> > > > >
> > > > > In a
> > > > >
> > > > > non-secure environment Authorizer is a blunt instrument either you
> > > > >
> > > > > end
> > > > >
> > > > > up
> > > > >
> > > > > blocking everyone or allowing everyone.
> > > > > I am asking to have server side that allow clients to create
> > > > >
> > > > > topics or
> > > > >
> > > > > not
> > > > >
> > > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > > replication-factor.
> > > > >
> > > > > -Harsha
> > > > >
> > > > > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com (
> > > ismaelj@gmail.com
> > > > > )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Harsha,
> > > > >
> > > > > Rogue clients can use the admin client to create topics and
> > > > >
> > > > > partitions.
> > > > >
> > > > > ACLs and policies can help in that case as well as this one.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha.
> > io
> > > (
> > > > > kafka@harsha.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > > Thanks for the KIP.
> > > > > "When server-side auto-creation is disabled, client-side
> > > > >
> > > > > auto-creation
> > > > >
> > > > > will try to use client-side configurations"
> > > > > If I understand correctly, this KIP is removing any server-side
> > > > >
> > > > > blocking
> > > > >
> > > > > client auto creation of topic?
> > > > > if so this will present potential issue of rogue client creating
> > > > >
> > > > > ton of
> > > > >
> > > > > topic-partitions and potentially bringing down the service for
> > > > >
> > > > > everyone
> > > > >
> > > > > or
> > > > >
> > > > > degrade the service itself.
> > > > > By reading the KIP its not clear to me that there is a clear way to
> > > > >
> > > > > block
> > > > >
> > > > > auto creation topics of all together from clients by server side
> > > > >
> > > > > config.
> > > > >
> > > > > Server side configs of default topic, partitions should take higher
> > > > > precedence and client shouldn't be able to create a topic with
> > > > >
> > > > > higher
> > > > >
> > > > > no.of
> > > > >
> > > > > partitions, replication than what server config specifies.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi all,
> > > > > I made some changes to the KIP. Hopefully this configuration change
> > > > >
> > > > > will
> > > > >
> > > > > make things a little clearer.
> > > > >
> > > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > > > https://cwiki.
> > > > > apache.org/confluence/display/KAFKA/ )
> > > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > > >
> > > > > Please let me know if you have any feedback or questions!
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache.
> > org (
> > > > > cmccabe@apache.org ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I think you bring up a good point. It would be better if we didn't
> > > > >
> > > > > ever
> > > > >
> > > > > have to set up client-side configuration for this feature, and
> > > > >
> > > > > KIP-464
> > > > >
> > > > > would let us skip this entirely.
> > > > >
> > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > > >
> > > > > Hi Mickael,
> > > > > I agree that KIP-464 works on newer brokers, but I was a bit
> > > > >
> > > > > worried
> > > > >
> > > > > how
> > > > >
> > > > > things would play out on older brokers that* do not *have KIP 464
> > > > >
> > > > > included.
> > > > >
> > > > > Is it enough to throw an error in this case when producer configs
> > > > >
> > > > > are
> > > > >
> > > > > not
> > > > >
> > > > > specified?
> > > > >
> > > > > I think the right thing to do would be to log an error message in
> > > > >
> > > > > the
> > > > >
> > > > > client. We will need to have that capability in any case, to cover
> > > > > scenarios like the client trying to auto-create a topic that they
> > > > >
> > > > > don't
> > > > >
> > > > > have permission to create. Or a client trying to create a topic on
> > > > >
> > > > > a
> > > > >
> > > > > broker
> > > > >
> > > > > so old that CreateTopicsRequest is not supported.
> > > > >
> > > > > The big downside to relying on KIP-464 is that it is a very recent
> > > > >
> > > > > feature
> > > > >
> > > > > -- so recent that it hasn't even made its way to any official
> > > > >
> > > > > Apache
> > > > >
> > > > > release. It's scheduled for the upcoming 2.4 release in a few
> > > > >
> > > > > months.
> > > > >
> > > > > So if you view this KIP as a step towards removing broker-side
> > > > > auto-create, you might want to support older brokers just to
> > > > >
> > > > > accelerate
> > > > >
> > > > > adoption, and hasten the day when we can finally flip broker-side
> > > > > auto-create to off (or even remove it entirely).
> > > > >
> > > > > I have to agree, though, that having client-side configurations for
> > > > >
> > > > > number
> > > > >
> > > > > of partitions and replication factor is messy. Maybe it would be
> > > > >
> > > > > worth
> > > > >
> > > > > it
> > > > >
> > > > > to restrict support to post-KIP-464 brokers, if we could avoid
> > > > >
> > > > > creating
> > > > >
> > > > > more configs.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> > > > >
> > > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > We can rely on KIP-464 which allows to omit the partition count or
> > > > > replication factor when creating a topic. In that case, the broker
> > > > >
> > > > > defaults
> > > > >
> > > > > are used.
> > > > >
> > > > > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Michael,
> > > > > That makes sense to me!
> > > > > To clarify, in the current state of the KIP, the producer does not
> > > > >
> > > > > rely
> > > > >
> > > > > on
> > > > >
> > > > > the broker to autocreate--if the broker's config is disabled, then
> > > > >
> > > > > the
> > > > >
> > > > > producer can autocreate on its own with a create topic request (the
> > > > >
> > > > > same
> > > > >
> > > > > type of request the admin client uses).
> > > > > However, if both configs are enabled, the broker will autocreate
> > > > >
> > > > > through
> > > > >
> > > > > a
> > > > >
> > > > > metadata request before the producer gets a chance. Of course, the
> > > > >
> > > > > way
> > > > >
> > > > > to
> > > > >
> > > > > avoid this, is to do as you suggested, and set
> > > > >
> > > > > the
> > > > >
> > > > > "allow_auto_topic_creation" field to false.
> > > > >
> > > > > I think the only thing we need to be careful with in this setup is
> > > > >
> > > > > without
> > > > >
> > > > > KIP 464, we can not use broker defaults for this topic. A user
> > > > >
> > > > > needs
> > > > >
> > > > > to
> > > > >
> > > > > specify the number of partition and replication factor in the
> > > > >
> > > > > config.
> > > > >
> > > > > An
> > > > >
> > > > > alternative to this is to have coded defaults for when these
> > > > >
> > > > > configs
> > > > >
> > > > > are
> > > > >
> > > > > unspecified, but it is not immediately apparent what these defaults
> > > > >
> > > > > should
> > > > >
> > > > > be.
> > > > >
> > > > > Thanks again for reading my KIP,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> > > > >
> > > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the response!
> > > > > In my opinion, it would be better if the producer did not rely at
> > > > >
> > > > > all
> > > > >
> > > > > on the broker auto create feature as this is what we're aiming to
> > > > > deprecate. When requesting metadata we can set the
> > > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > > creation. Then if the topic is not existing, send a
> > > > >
> > > > > CreateTopicRequest.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Currently the way it is implemented, the broker auto-creation
> > > > >
> > > > > configuration
> > > > >
> > > > > takes precedence. The producer will not use the CreateTopics
> > > > >
> > > > > request.
> > > > >
> > > > > (Technically it can--but the topic will already be created
> > > > >
> > > > > through
> > > > >
> > > > > the
> > > > >
> > > > > broker, so it will never try to create the topic.) It is possible
> > > > >
> > > > > to
> > > > >
> > > > > change this however, and I'd be happy to
> > > > >
> > > > > discuss
> > > > >
> > > > > the
> > > > >
> > > > > benefits of this alternative.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> > > > >
> > > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP!
> > > > >
> > > > > In case auto creation is enabled on both the client and server,
> > > > >
> > > > > will
> > > > >
> > > > > the producer still use the AdminClient (CreateTopics request)
> > > > >
> > > > > to
> > > > >
> > > > > create topics? and not rely on the broker auto create. I'm
> > > > >
> > > > > guessing the
> > > > >
> > > > > answer is yes but can you make it explicit.
> > > > >
> > > > > Thank you
> > > > >
> > > > > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi,
> > > > > Just a friendly reminder to take a look at this KIP if you
> > > > >
> > > > > have
> > > > >
> > > > > the
> > > > >
> > > > > time.
> > > > >
> > > > > I was thinking about broker vs. client default precedence,
> > > > >
> > > > > and I
> > > > >
> > > > > think it
> > > > >
> > > > > makes sense to keep the broker as the default used when both
> > > > >
> > > > > client-side
> > > > >
> > > > > and broker-side defaults are configured. The idea is that
> > > > >
> > > > > there
> > > > >
> > > > > would be
> > > > >
> > > > > pretty clear documentation, and that many systems with
> > > > >
> > > > > configurations
> > > > >
> > > > > that
> > > > >
> > > > > the client could not change would likely have the auto-create
> > > > >
> > > > > default
> > > > >
> > > > > off.
> > > > >
> > > > > (In cloud for example).
> > > > >
> > > > > It also seems like in most cases, the consumer config
> > > > > ' allow. auto. create. topics ( http://allow.auto.create.topics/ ) '
> > > was
> > > > > created to actually prevent
> > > > >
> > > > > the
> > > > >
> > > > > creation
> > > > >
> > > > > of
> > > > >
> > > > > topics, so the loss of creation functionality will not be a
> > > > >
> > > > > big
> > > > >
> > > > > problem.
> > > > >
> > > > > I'm happy to discuss any other compatibility problems or
> > > > >
> > > > > components
> > > > >
> > > > > of
> > > > >
> > > > > this KIP.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > I was looking at this KIP again, and there is a decision I
> > > > >
> > > > > made
> > > > >
> > > > > that I
> > > > >
> > > > > think is worth discussing.
> > > > >
> > > > > In the case where both the broker and producer's
> > > > > 'auto.create.topics.enable' are set to true, we have to
> > > > >
> > > > > choose
> > > > >
> > > > > either
> > > > >
> > > > > the
> > > > >
> > > > > broker configs or the producer configs for the replication
> > > > > factor/partitions.
> > > > >
> > > > > Currently, the decision is to have the broker defaults take
> > > > >
> > > > > precedence.
> > > > >
> > > > > (It is easier to do this in the implementation.) It also
> > > > >
> > > > > makes
> > > > >
> > > > > some
> > > > >
> > > > > sense
> > > > >
> > > > > for this behavior to take precedence since this behavior
> > > > >
> > > > > already
> > > > >
> > > > > occurs as
> > > > >
> > > > > the default.
> > > > >
> > > > > However, I was wondering if it would be odd for those who
> > > > >
> > > > > can
> > > > >
> > > > > only
> > > > >
> > > > > see
> > > > >
> > > > > the
> > > > >
> > > > > producer side to set configs for replication
> > > > >
> > > > > factor/partitions
> > > > >
> > > > > and
> > > > >
> > > > > see
> > > > >
> > > > > different results. Currently the documentation for the
> > > > >
> > > > > config
> > > > >
> > > > > states
> > > > >
> > > > > that
> > > > >
> > > > > the config values are only used when the broker config is
> > > > >
> > > > > not
> > > > >
> > > > > enabled,
> > > > >
> > > > > but
> > > > >
> > > > > this might not always be clear to the user. Changing the
> > > > >
> > > > > code
> > > > >
> > > > > to
> > > > >
> > > > > have
> > > > >
> > > > > the
> > > > >
> > > > > producer's configurations take precedence is possible, but
> > > > >
> > > > > I
> > > > >
> > > > > was
> > > > >
> > > > > wondering
> > > > >
> > > > > what everyone thought.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Just a quick update--
> > > > >
> > > > > It seems that enabling both the broker and producer
> > > > >
> > > > > configs
> > > > >
> > > > > works
> > > > >
> > > > > fine,
> > > > >
> > > > > except that the broker configurations for partitions,
> > > > >
> > > > > replication
> > > > >
> > > > > factor
> > > > >
> > > > > take precedence.
> > > > > I don't know if that is something we would want to
> > > > >
> > > > > change, but
> > > > >
> > > > > I'll be
> > > > >
> > > > > updating the KIP for now to reflect this. Perhaps we would
> > > > >
> > > > > want to
> > > > >
> > > > > add more
> > > > >
> > > > > to the documentation of the the producer configs to
> > > > >
> > > > > clarify.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Colin,
> > > > >
> > > > > Thanks for looking at the KIP. I can definitely add to
> > > > >
> > > > > the
> > > > >
> > > > > title
> > > > >
> > > > > to
> > > > >
> > > > > make
> > > > >
> > > > > it more clear.
> > > > >
> > > > > It makes sense that both configurations could be turned
> > > > >
> > > > > on
> > > > >
> > > > > since
> > > > >
> > > > > there
> > > > >
> > > > > are many cases where the user can not control the
> > > > >
> > > > > server-side
> > > > >
> > > > > configurations. I was a little concerned about how both
> > > > >
> > > > > interacting
> > > > >
> > > > > would
> > > > >
> > > > > work out -- if there would be to many requests for new
> > > > >
> > > > > topics,
> > > > >
> > > > > for
> > > > >
> > > > > example.
> > > > >
> > > > > But it since it does make sense to allow both
> > > > >
> > > > > configurations
> > > > >
> > > > > enabled, I
> > > > >
> > > > > will test out some scenarios and I'll change the KIP to
> > > > >
> > > > > support
> > > > >
> > > > > this.
> > > > >
> > > > > I also agree with documentation about distinguishing the
> > > > >
> > > > > differences. I
> > > > >
> > > > > was having some trouble with the wording but I like the
> > > > >
> > > > > phrases
> > > > >
> > > > > "server-side" and "client-side." That's a good
> > > > >
> > > > > distinction I
> > > > >
> > > > > can
> > > > >
> > > > > use
> > > > >
> > > > > when
> > > > >
> > > > > describing.
> > > > >
> > > > > I'll try to update the KIP soon keeping everyone's input
> > > > >
> > > > > in
> > > > >
> > > > > mind.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> > > > >
> > > > > cmccabe@ apache. org ( cmccabe@apache.org )
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP. This seems like a good step towards
> > > > >
> > > > > removing
> > > > >
> > > > > server-side topic auto-creation.
> > > > >
> > > > > We should add included "client-side" to the title of
> > > > >
> > > > > the KIP
> > > > >
> > > > > somewhere,
> > > > >
> > > > > to make it clear that we're talking about client-side
> > > > >
> > > > > auto
> > > > >
> > > > > creation.
> > > > >
> > > > > The KIP says:
> > > > >
> > > > > In order to automatically create topics with the
> > > > >
> > > > > producer, the
> > > > >
> > > > > producer's
> > > > >
> > > > > auto.create.topics.enable config must be set to true
> > > > >
> > > > > and
> > > > >
> > > > > the
> > > > >
> > > > > broker
> > > > >
> > > > > config should be set to false
> > > > >
> > > > > From a user's point of view, this seems
> > > > >
> > > > > counter-intuitive.
> > > > >
> > > > > In
> > > > >
> > > > > order to
> > > > >
> > > > > auto-create topics the broker's
> > > > >
> > > > > auto.create.topics.enable
> > > > >
> > > > > config
> > > > >
> > > > > should be
> > > > >
> > > > > set to false? It seems like the server-side
> > > > >
> > > > > auto-create is
> > > > >
> > > > > unrelated to
> > > > >
> > > > > the client-side auto-create. We could have both turned
> > > > >
> > > > > on
> > > > >
> > > > > (and
> > > > >
> > > > > I'm
> > > > >
> > > > > sure
> > > > >
> > > > > that in the real world, people will try this
> > > > >
> > > > > configuration...)
> > > > >
> > > > > There's no
> > > > >
> > > > > reason not to support this, I think.
> > > > >
> > > > > We should add some documentation explaining the
> > > > >
> > > > > difference
> > > > >
> > > > > between
> > > > >
> > > > > server-side and client-side auto-creation. Without
> > > > >
> > > > > documentation,
> > > > >
> > > > > an admin
> > > > >
> > > > > might think that they had disabled all forms of
> > > > >
> > > > > auto-creation by
> > > > >
> > > > > setting
> > > > >
> > > > > the -side setting to false-- but this is not the case,
> > > > >
> > > > > of
> > > > >
> > > > > course.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > > >
> > > > > Hi Dhruvil,
> > > > >
> > > > > Thanks for reading the KIP!
> > > > > That was the general idea for deprecation. We would
> > > > >
> > > > > log a
> > > > >
> > > > > warning
> > > > >
> > > > > when the
> > > > >
> > > > > config is enabled on the broker.
> > > > > I also believe that there would be a change to
> > > > >
> > > > > documentation.
> > > > >
> > > > > If there is anything else that should be done, please
> > > > >
> > > > > let
> > > > >
> > > > > me
> > > > >
> > > > > know!
> > > > >
> > > > > Justine
> > > > >
> > > > > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> > > > >
> > > > > dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP, this is great!
> > > > >
> > > > > Could you add some more information about what
> > > > >
> > > > > deprecating
> > > > >
> > > > > the
> > > > >
> > > > > broker
> > > > >
> > > > > configuration means? Would we log a warning in the
> > > > >
> > > > > logs
> > > > >
> > > > > when
> > > > >
> > > > > auto
> > > > >
> > > > > topic
> > > > >
> > > > > creation is enabled on the broker, for example?
> > > > >
> > > > > Thanks,
> > > > > Dhruvil
> > > > >
> > > > > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> > > > >
> > > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > I'd like to start a discussion thread for KIP-487. This KIP plans
> > > > >
> > > > > to
> > > > >
> > > > > deprecate the current system of
> > > > >
> > > > > auto-creating
> > > > >
> > > > > topics
> > > > >
> > > > > through requests to the metadata and give the
> > > > >
> > > > > producer the
> > > > >
> > > > > ability to
> > > > >
> > > > > automatically create topics instead.
> > > > >
> > > > > More information can be found here:
> > > > >
> > > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > > > https://cwiki.
> > > > > apache.org/confluence/display/KAFKA/ )
> > > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > > >
> > > > > Thank you,
> > > > > Justine Olshan
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Jun Rao <ju...@confluent.io>.

Hi, Justine,

Thanks for the KIP. Overall, it seems to be a good improvement.

However, I think Harsha's point seems reasonable. We had
auto.create.topics.enable config on the broker to allow admins to disable
topic creation from the producer/consumer clients before we had the
security feature. The need for that config is reduced with the security
feature, but may still be present since not all places have security
enabled. It's true that a non-secured environment is vulnerable to some
additional attacks, but producer/consumer are the most common way for a
user to interact with the broker. So, keeping that config for backward
compatibility could still be useful if it's not introducing too much effort
or extra confusion.

Here is a one potential alternative way that I was thinking. We add a new
field in the CreateTopicRequest to indicate whether it's from the producer
or not. If auto.create.topics.enable is false, CreateTopicRequest from the
producer will be rejected. We probably don't need to introduce the new
config (which seems a bit hard to explain) in the producer. Instead, the
new producer always uses MetadataRequest with AllowAutoTopicCreation set to
false to get the metadata and if the metadata is not present, send the
new CreateTopicRequest
(assuming the broker supports it) to try to create the topic automatically.
Whether the creation is allowed or not will be determined by the broker.
This will make the behavior backward compatible and we can still achieve
the main goal of the KIP, which is not relying on MetadataRequest for topic
creation. What do you think?

Thanks,

Jun

On Thu, Aug 8, 2019 at 1:34 AM M. Manna <ma...@gmail.com> wrote:

> Hi,
>
> If I may, perhaps you could simplify everything by using only
> 'auto.create.topics.enable' as a value along with true. In other words, the
> public interfaces section should only have [true,auto.create.topics.enable,
> false].
>
> The reason for this is that auto.create.topics.enable is already known to
> users as a "Server-SIde" config. So all you are saying is
>
> a) To avoid day 1 impact, it will follow whatever auto.create.topics.enable
> value is set.
> b) False means - no client side topic creation
> c) True means client side topic creation.
>
> It saves creating 2 more new strings :). But not too expensive anyway.
>
> Also, when you deprecate auto.create.topics.enable - you must provide
> sufficient logic to ensure that things like rolling upgrade doesn't
> temporarily break anything. I apologise if you have already accounted for
> this, but wanted to mention since I didn't notice this on the KIP.
>
> Let me know how this sounds.
>
> Regards,
>
> On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jo...@confluent.io> wrote:
>
> > Hi Harsha,
> >
> > I think my message may have gotten lost in all the others.
> >
> > Two of the goals of this KIP are to 1) allow auto-creation on specific
> > clients when the broker default is false and 2) eventually replace the
> > broker config.
> >
> > In order to accomplish these two goals, we need the producer to be able
> to
> > create topics despite the broker config. (How can we replace a function
> > when we rely on it?)
> > I think at this point we have a fundamental disagreement in what we
> should
> > allow the producer to do.
> > In my previous message I mentioned a config that would allow for the
> broker
> > to prevent producer auto-creation. (It would be disabled by default.) It
> > would fix your issue for now, but could lead to more complications later.
> >
> > Thank you,
> > Justine
> >
> >
> > On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz>
> wrote:
> > >
> > > > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> > > >
> > > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmccabe@apache.org
> >
> > > > wrote:
> > > >
> > > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> > > >
> > > > Hi Colin,
> > > > "Hmm... I'm not sure I follow. Users don't have to build their own
> > > > tooling, right? They can use any of the shell scripts that we've
> > shipped
> > > in
> > > > the last few releases. For example, if any of your users run it, this
> > > shell
> > > > script will delete all of the topics from your non-security-enabled
> > > > cluster:
> > > >
> > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > --bootstrap-server localhost:9092 --delete
> > > > --topic
> > > >
> > > > They will need to fill in the correct bootstrap servers list, of
> > course,
> > > > not localhost. This deletion script will work on some pretty old
> > brokers,
> > > > even back to the 0.10 releases. It seems a little odd to trust your
> > users
> > > > with this power, but not trust them to avoid changing a particular
> > > > configuration key."
> > > >
> > > > The above will blocked by the server if we set delete.topic.enable to
> > > > false and thats exactly what I am asking for.
> > > >
> > > > Hi Harsha,
> > > >
> > > > I was wondering if someone was going to bring up that configuration
> :)
> > > >
> > > > it's an interesting complication, but globally disabling topic
> deletion
> > > is
> > > > not very practical for most use-cases.
> > > >
> > > > In any case, there are plenty of other bad things that users with
> full
> > > > permissions can do that aren't blocked by any server configuration.
> For
> > > > example, they can delete every record in every topic. I can write a
> > > script
> > > > for that too, and there's no server configuration you can set to
> > disable
> > > > it. Or I could simply create hundreds of thousands of topics, until
> > > cluster
> > > > performance becomes unacceptable (this will be even more of a problem
> > if
> > > > someone configured delete.topic.enable as false). Or publish bad data
> > to
> > > > every topic, etc. etc.
> > > >
> > > > The point I'm trying to make here is that you can't rely on these
> kind
> > of
> > > > server-side configurations for security. At most, they're a way to
> set
> > up
> > > > certain very simple policies. But the policies are so simple that
> > they're
> > > > hardly ever useful any more.
> > > >
> > > > For example, if the problem you want to solve is that you want a user
> > to
> > > > only be able to create 50 topics and not delete anyone else's topics,
> > you
> > > > can solve that with a CreateTopicsPolicy that limits the number of
> > > topics,
> > > > and some ACLs. There's no combination of auto.create.topics.enable
> and
> > > > delete.topic.enable that will help here.
> > > >
> > > > Hi Colin,
> > > >
> > > >             Well you gave the example that a user can delete the
> topics
> > > > just by running that script  :).
> > > >
> > > > I understand there are open APIs in Kafka and can lead to rogue
> clients
> > > > taking advantage of it without proper security in place.
> > > >
> > > > What I am asking so far in this thread is , this KIP is changing the
> > > > producer behavior and its not backward compatible.
> > > >
> > > > The change is backwards compatible. The default will still be
> > server-side
> > > > topic auto-creation, just like now.
> > > >
> > > You will have to specifically change the producer config to get the new
> > > > behavior.
> > > >
> > >
> > >
> > > I disagree.  Today server can turn off the topic creation  neither
> > producer
> > > or consumer can create a topic. With this KIP , producer can create a
> > topic
> > > by turning on client side config when server side config is turned off.
> > >
> > >
> > > We can still achieve
> > > > the main goal of the KIP which is to change MetadataRequest  creating
> > > > topics and send a CreateTopicRequest from Producer and also keep the
> > > server
> > > > side config to have precedence.  This KIP originally written to have
> > > server
> > > > side preference and there is not much explanation why it changed to
> > have
> > > > Producer side preference.
> > > >
> > > > Arguing that AdminClient can do that and so we are going to make
> > Producer
> > > > do the same doesn't make sense.
> > > >
> > > > "The downside is that if we wanted to check a server side
> configuration
> > > > before sending the create topics request, the code would be more
> > complex.
> > > > The behavior would also not be consistent with how topic
> auto-creation
> > is
> > > > handled in Kafka Streams."
> > > >
> > > > I am not sure why you need to check server side configuration before
> > > > sending create topics request. A user enables producer side config to
> > > > create topics.
> > > > Producer sends a request to the broker and if the broker has
> > > > auto.topic.create.enable to true (default) it will allow creation of
> > > > topics. If it set to false it returns error back to the client.
> > > >
> > > > auto.topic.create.enable has never affected CreateTopicsRequest. If
> you
> > > > submit a CreateTopicsRequest and you are authorized, a topic will be
> > > > created, regardless of what the value of auto.topic.create.enable is.
> > > This
> > > > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?)
> > > >
> > > > I don't see how this behavior will be different in Kafka streams. By
> > > > default server allows the topic creation and with this KIP, It will
> > only
> > > > allow creation of topic when both producer and server side are turned
> > on.
> > > > Its exactly the same behavior in KIP-361.
> > > >
> > > > I suggest running a streams job as a test. It creates the topics it
> > needs
> > > > using AdminClient. The value of auto.topic.create.enable is not
> > relevant.
> > > > Whether it is true or false, the topics will still be created.
> > > >
> > > > "In general, it would be nice if we could keep the security and
> access
> > > > control model simple and not introduce a lot of special cases and
> > > > exceptions. Kafka has basically converged on using ACLs and
> > > > CreateTopicPolicy classes to control who can create topics and where.
> > > > Adding more knobs seems like a step backwards, especially when the
> > > proposed
> > > > knobs don't work consistently across components, and don't provide
> true
> > > > security." This is not about access control at all. Shipping sane
> > > defaults
> > > > should be prioritized.
> > > >
> > > > I don't think the default is really the question here. I think
> everyone
> > > > agrees that the default for client-side auto-topic creation should be
> > > off.
> > > >
> > > > My point goes back to KIP-361 why we kept the server side config to
> > have
> > > > preference. We can keep bringing back the discussion of security
> > policies
> > > > but a producer client overiding server side setting is not just
> > security
> > > > policy issue .
> > > >
> > > > "Producer client can override server side setting and not consumer
> and
> > > > also producer can only override creation of topic but no the
> > replication
> > > > and partitions. " This doesn't make sense to me and why we want to
> > > > introduce such a confusing behavior.
> > > >
> > > > The scenarios that you're describing (such as dealing with a poorly
> > > > configured client that tries to create topics it should not) do seem
> to
> > > be
> > > > about access control.
> > > >
> > > > We keep talking about CreateTopicPolicy and yet we don't have default
> > one
> > > > and asking every user of Kafka implement their own doesn't make sense
> > > here.
> > > >
> > > > The point of CreateTopicPolicy is exactly that you can implement your
> > > own,
> > > > though. It's a way of customizing what the policy is in your specific
> > > > cluster.
> > > >
> > > > I agree that most people don't want to write a custom policy. But
> most
> > of
> > > > those people are satisfied with just setting up ACLs to set up simple
> > > > policies like this user can create topics, that user can't, etc.
> That's
> > > why
> > > > I suggested adding support for ACLs to insecure clusters might be
> > > helpful.
> > > >
> > > > Also, just as a side note, wouldn't it would be impossible for us to
> > > > specify a default CreateTopicsPolicy that did anything at this point
> > > anyway
> > > > without breaking backwards compatibility? Maybe I'm misunderstanding
> > the
> > > > proposal.
> > > >
> > > > I am asking about exact behavior that we shipped for consumer side
> > > https:/
> > > > / cwiki. apache. org/ confluence/ display/ KAFKA/
> > > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > > (
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > > )
> > > >
> > > > I still didn't get any response why this behavior shouldn't be
> exactly
> > > > like Kafka consumer and why do we want to have producer to overrider
> > > server
> > > > side config and while not allowing consumer to do so. We are not even
> > > > allowing the same contract and this will cause more confusion from
> the
> > > > users standpoint.
> > > >
> > > > Hmm. The consumer already has a way to override the server side
> > > > configuration. See KIP-361: Add Consumer Configuration to Disable
> Auto
> > > > Topic Creation. This lets the consumer skip auto-creating topics,
> even
> > if
> > > > auto-creation is enabled on the broker.
> > > >
> > > > Yes that's what I am saying and it doesn't allow topic creation if
> the
> > > > server side auto-creation is disabled and in consumer its enabled.
>  I
> > > > would like to see the exact behavior for producer as stated in
> KIP-361.
> > > >
> > > > To be fair, the KIP should probably discuss why we don't want to
> > > implement
> > > > client-side topic creation in the consumer in "rejected
> alternatives."
> > > > Maybe Justine can add more context here in the KIP.
> > > >
> > > > The last time we talked about this, the reasoning was that we wanted
> to
> > > > eventually get rid of consumer-side auto-topic creation entirely, and
> > so
> > > it
> > > > wasn't worth implementing an improved way of doing it. Producer side
> > auto
> > > > topic-creation, in contrast, will probably stick around, although
> we'd
> > > like
> > > > to deprecate and remove the problematic server-side mechanism over
> > time.
> > > >
> > > > We should implement the producer side topic creation with proper
> create
> > > > topic request  and I am in favor of this KIP.
> > > >
> > > > All I am asking is don't make the producer to override server side
> > config
> > > > and keep it similar to KIP-361 just like consumer, it honors what set
> > on
> > > > server side which takes the precedence.   Changing this behavior in
> > > > producer is backward incompatible and will catch users by surprise.
> > > >
> > > > All arguments for enforcing producer side overriding config can still
> > be
> > > > achieved. Server side auto creation turned on by default and  any new
> > > > producer client setting their auto creation config to true will
> create
> > > > topics in default behavior and any user who set server side to false
> > and
> > > > upgrades to latest kafka with this KIP will not see any unwanted
> > behavior
> > > > from clients.  IMO this is not a unreasonable request on this KIP and
> > > this
> > > > requested behavior is exactly what the KIP initially proposed.
> > > >
> > > > Thanks,
> > > >
> > > > Harsha
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache. org (
> > > > cmccabe@apache.org ) > wrote:
> > > >
> > > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > > >
> > > > Not sure how the AdminClient applies here, It is an external API and
> is
> > > > not part of KafkaProducer so any user who updates to latest version
> of
> > > > Kafka with this feature get to create the topics.
> > > > They have to build a tooling around AdminClient allow themselves to
> > > >
> > > > create
> > > >
> > > > topics.
> > > >
> > > > Hi Harsha,
> > > >
> > > > Hmm... I'm not sure I follow. Users don't have to build their own
> > > tooling,
> > > > right? They can use any of the shell scripts that we've shipped in
> the
> > > last
> > > > few releases. For example, if any of your users run it, this shell
> > script
> > > > will delete all of the topics from your non-security-enabled cluster:
> > > >
> > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > > --bootstrap-server localhost:9092 --delete
> > > > --topic
> > > >
> > > > They will need to fill in the correct bootstrap servers list, of
> > course,
> > > > not localhost. This deletion script will work on some pretty old
> > brokers,
> > > > even back to the 0.10 releases. It seems a little odd to trust your
> > users
> > > > with this power, but not trust them to avoid changing a particular
> > > > configuration key.
> > > >
> > > > There is no behavior in Kafka producer that allowed users to delete
> the
> > > > topics or delete the records. So citing them as an example doesn't
> > makes
> > > > sense in this context.
> > > >
> > > > I think Kafka Streams is relevant here. After all, it's software that
> > we
> > > > ship as part of the official Kafka release. And it auto-creates
> topics
> > > even
> > > > when auto.create.topics.enable is set to false on the broker.
> > > >
> > > > I think that auto.create.topics.enable was never intended as a
> security
> > > > setting to control access. It was intended as a way to disable the
> > > > broker-side auto-create feature specifically, because there were some
> > > > problems with that specific feature. Broker-side auto-creation is
> > > > frustrating because it's on by default, and because it can
> auto-create
> > > > topics even if the producer or consumer didn't explicitly ask for
> that.
> > > > Neither of those problems applies to this KIP: producers have to
> > > > specifically opt in, and it won't be on by default. Basically, we
> think
> > > > that client-side auto-creation is actually a lot better-- hence this
> > KIP
> > > > :)
> > > >
> > > > But there is
> > > > a functionality which allowed creation of topics if they don't exist
> in
> > > >
> > > > the
> > > >
> > > > cluster and this behavior could be controlled by a config on the
> server
> > > > side. Now with this KIP we are allowing producer to make that
> decision
> > > > without any gateway on the server via configs. Any user who is not
> > aware
> > > >
> > > > of
> > > >
> > > > such changes
> > > > can accidentally create these topics and we are essentially removing
> a
> > > > config that exists in brokers today to block this accidental creation
> > and
> > > > allowing clients to take control.
> > > >
> > > > Again, I hope I'm not misinterpreting, but I don't see how this can
> be
> > > > turned on accidentally. The user would have to specifically turn this
> > on
> > > in
> > > > the producer by setting the configuration key.
> > > >
> > > > I still didn't get any positives of not having server side configs?
> if
> > > you
> > > > want to turn them on and allow any client to create topics set the
> > > default
> > > > to true
> > > > and allow users who doesn't want to have this feature let them turn
> > them
> > > > off. It will be the exact behavior as it is today , as far as
> producer
> > is
> > > > concerned. I am not
> > > > understanding why not having server side configs to gateways such a
> > hard
> > > > requirement and this behavior exists today. As far I am concerned
> this
> > > > breaks the backward compatibility.
> > > >
> > > > The downside is that if we wanted to check a server side
> configuration
> > > > before sending the create topics request, the code would be more
> > complex.
> > > > The behavior would also not be consistent with how topic
> auto-creation
> > is
> > > > handled in Kafka Streams.
> > > >
> > > > In general, it would be nice if we could keep the security and access
> > > > control model simple and not introduce a lot of special cases and
> > > > exceptions. Kafka has basically converged on using ACLs and
> > > > CreateTopicPolicy classes to control who can create topics and where.
> > > > Adding more knobs seems like a step backwards, especially when the
> > > proposed
> > > > knobs don't work consistently across components, and don't provide
> true
> > > > security.
> > > >
> > > > Maybe we should support an insecure mode where there are still users
> > and
> > > > ACLs. That would help people who don't want to set up Kerberos, etc.
> > but
> > > > still want to protect against misconfigured clients or accidents.
> > Hadoop
> > > > has something like this, and I think it was useful. NFS also
> supported
> > > > (supports?) a mode where you just pass whatever user ID you want and
> > the
> > > > system believes you. These things clearly don't protect against
> > malicious
> > > > users, but they can help set up policies when needed.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache. org (
> > > > cmccabe@apache.org ) > wrote:
> > > >
> > > > Hi Harsha,
> > > >
> > > > Thanks for taking a look.
> > > >
> > > > I'm not sure I follow the discussion about AdminClient.
> > > >
> > > > KafkaAdminClient
> > > >
> > > > has been around for about 2 years at this point as a public class.
> > > >
> > > > There
> > > >
> > > > are many programs that use it to automatically create topics. Kafka
> > > > Streams does this, for example. If any of your users run Kafka
> > > >
> > > > Streams,
> > > >
> > > > they will be auto-creating topics, regardless of what setting you use
> > > >
> > > > for
> > > >
> > > > auto.create.topics.enable.
> > > >
> > > > A big part of the annoyance of auto-topic creation right now is that
> > > >
> > > > it is
> > > >
> > > > on by default. The new configuration proposed by KIP-487 wouldn't be.
> > > > Users would have to explicitly opt in to the new behavior of
> > > >
> > > > client-side
> > > >
> > > > topic creation. If you run without security, you're already putting a
> > > >
> > > > huge
> > > >
> > > > amount of trust in your users. For example, you trust them not to
> > > >
> > > > delete
> > > >
> > > > records with the kafka-delete-records. sh (
> > http://kafka-delete-records.
> > > > sh/
> > > > ) command, or delete topics with kafka-topics. sh (
> > http://kafka-topics.
> > > > sh/
> > > > ). Trusting them not to set a certain config value seems minor in
> > > > comparison, right?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > > >
> > > > Hi,
> > > > Even with policies one needs to implement that, so for every
> > > >
> > > > user who
> > > >
> > > > doesn't want a producer to create topics or have limits around
> > > >
> > > > partitions &
> > > >
> > > > replication factor they have to implement a policy. The KIP is
> changing
> > > > the behavior , it might not be introducing
> > > >
> > > > the
> > > >
> > > > new functionality but it will enable producers to override the create
> > > >
> > > > topic
> > > >
> > > > config settings on the broker. What I am asking for to provide a
> > > >
> > > > config
> > > >
> > > > that will disable auto creation of topics and if its enabled set some
> > > >
> > > > sane
> > > >
> > > > defaults so that clients can create a topic with in those limits. I
> > > >
> > > > don't
> > > >
> > > > see how this not related to this KIP.
> > > > If the server config options as I suggested doesn't interest
> > > >
> > > > you at
> > > >
> > > > least have a default CreateTopicPolices in place.
> > > > To give an example, In our environment we disable the
> > > > auto.create.topic.enable and force users to go through a centralized
> > > > service as we want capture more details about the topic creation and
> > > > requirements. With this KIP, a producer can create a topic with no
> > > >
> > > > bounds.
> > > >
> > > > Another example max.message.size we define that at cluster level
> > > >
> > > > and one
> > > >
> > > > can override max.messsage.size at topic level, any misconfiguration
> > > >
> > > > at
> > > >
> > > > this
> > > >
> > > > will cause service degradation. Its not always about the rogue
> > > >
> > > > clients,
> > > >
> > > > users can easily misconfigure and can cause an outage. Again we can
> > talk
> > > > about CreateTopicPolicy but without having a
> > > >
> > > > default
> > > >
> > > > implementation and asking everyone to implement their own while
> > > >
> > > > changing
> > > >
> > > > the behavior in producer doesn't make sense to me.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me.
> uk (
> > > > ismael@juma.me.uk ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Harsha,
> > > >
> > > > I mentioned policies and the authorizer. For example, with
> > > > CreateTopicPolicy, you can implement the limits you describe. If
> > > >
> > > > you
> > > >
> > > > have
> > > >
> > > > ideas of how that should be improved, please submit a KIP. My
> > > >
> > > > point is
> > > >
> > > > that
> > > >
> > > > this KIP is not introducing any new functionality with regards to
> > > >
> > > > what
> > > >
> > > > rogue clients can do. It's using the existing protocol that is
> > > >
> > > > already
> > > >
> > > > exposed via the AdminClient. So, I don't think we need to address
> > > >
> > > > it in
> > > >
> > > > this KIP. Does that make sense?
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> > > >
> > > > kafka@ harsha. io ( kafka@harsha.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Ismael,
> > > > Sure AdminClient can do that and we should've shipped a config or
> > > >
> > > > use
> > > >
> > > > the
> > > >
> > > > existing one to block that. Not all users are yet to upgrade to
> > > >
> > > > AdminClient
> > > >
> > > > and start using that to cause issues yet. In shared environment we
> > > >
> > > > should
> > > >
> > > > allow server to set sane defaults and not allow every client to go
> > > >
> > > > ahead
> > > >
> > > > create random no.of topic/partitions and replication factor. Even
> > > >
> > > > if
> > > >
> > > > the
> > > >
> > > > users want to allow topic creation proposed in the KIP , it makes
> > > >
> > > > sense to
> > > >
> > > > have some guards against the no.of partitions and replication
> > > >
> > > > factor.
> > > >
> > > > Authorizer is not always an answer to block requests and having
> > > >
> > > > users
> > > >
> > > > set
> > > >
> > > > server side configs to protect a multi-tenant environment is
> > > >
> > > > required.
> > > >
> > > > In a
> > > >
> > > > non-secure environment Authorizer is a blunt instrument either you
> > > >
> > > > end
> > > >
> > > > up
> > > >
> > > > blocking everyone or allowing everyone.
> > > > I am asking to have server side that allow clients to create
> > > >
> > > > topics or
> > > >
> > > > not
> > > >
> > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > replication-factor.
> > > >
> > > > -Harsha
> > > >
> > > > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com (
> > ismaelj@gmail.com
> > > > )
> > > >
> > > > wrote:
> > > >
> > > > Harsha,
> > > >
> > > > Rogue clients can use the admin client to create topics and
> > > >
> > > > partitions.
> > > >
> > > > ACLs and policies can help in that case as well as this one.
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha.
> io
> > (
> > > > kafka@harsha.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > > Thanks for the KIP.
> > > > "When server-side auto-creation is disabled, client-side
> > > >
> > > > auto-creation
> > > >
> > > > will try to use client-side configurations"
> > > > If I understand correctly, this KIP is removing any server-side
> > > >
> > > > blocking
> > > >
> > > > client auto creation of topic?
> > > > if so this will present potential issue of rogue client creating
> > > >
> > > > ton of
> > > >
> > > > topic-partitions and potentially bringing down the service for
> > > >
> > > > everyone
> > > >
> > > > or
> > > >
> > > > degrade the service itself.
> > > > By reading the KIP its not clear to me that there is a clear way to
> > > >
> > > > block
> > > >
> > > > auto creation topics of all together from clients by server side
> > > >
> > > > config.
> > > >
> > > > Server side configs of default topic, partitions should take higher
> > > > precedence and client shouldn't be able to create a topic with
> > > >
> > > > higher
> > > >
> > > > no.of
> > > >
> > > > partitions, replication than what server config specifies.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi all,
> > > > I made some changes to the KIP. Hopefully this configuration change
> > > >
> > > > will
> > > >
> > > > make things a little clearer.
> > > >
> > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > > https://cwiki.
> > > > apache.org/confluence/display/KAFKA/ )
> > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > >
> > > > Please let me know if you have any feedback or questions!
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache.
> org (
> > > > cmccabe@apache.org ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think you bring up a good point. It would be better if we didn't
> > > >
> > > > ever
> > > >
> > > > have to set up client-side configuration for this feature, and
> > > >
> > > > KIP-464
> > > >
> > > > would let us skip this entirely.
> > > >
> > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > >
> > > > Hi Mickael,
> > > > I agree that KIP-464 works on newer brokers, but I was a bit
> > > >
> > > > worried
> > > >
> > > > how
> > > >
> > > > things would play out on older brokers that* do not *have KIP 464
> > > >
> > > > included.
> > > >
> > > > Is it enough to throw an error in this case when producer configs
> > > >
> > > > are
> > > >
> > > > not
> > > >
> > > > specified?
> > > >
> > > > I think the right thing to do would be to log an error message in
> > > >
> > > > the
> > > >
> > > > client. We will need to have that capability in any case, to cover
> > > > scenarios like the client trying to auto-create a topic that they
> > > >
> > > > don't
> > > >
> > > > have permission to create. Or a client trying to create a topic on
> > > >
> > > > a
> > > >
> > > > broker
> > > >
> > > > so old that CreateTopicsRequest is not supported.
> > > >
> > > > The big downside to relying on KIP-464 is that it is a very recent
> > > >
> > > > feature
> > > >
> > > > -- so recent that it hasn't even made its way to any official
> > > >
> > > > Apache
> > > >
> > > > release. It's scheduled for the upcoming 2.4 release in a few
> > > >
> > > > months.
> > > >
> > > > So if you view this KIP as a step towards removing broker-side
> > > > auto-create, you might want to support older brokers just to
> > > >
> > > > accelerate
> > > >
> > > > adoption, and hasten the day when we can finally flip broker-side
> > > > auto-create to off (or even remove it entirely).
> > > >
> > > > I have to agree, though, that having client-side configurations for
> > > >
> > > > number
> > > >
> > > > of partitions and replication factor is messy. Maybe it would be
> > > >
> > > > worth
> > > >
> > > > it
> > > >
> > > > to restrict support to post-KIP-464 brokers, if we could avoid
> > > >
> > > > creating
> > > >
> > > > more configs.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> > > >
> > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > We can rely on KIP-464 which allows to omit the partition count or
> > > > replication factor when creating a topic. In that case, the broker
> > > >
> > > > defaults
> > > >
> > > > are used.
> > > >
> > > > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Michael,
> > > > That makes sense to me!
> > > > To clarify, in the current state of the KIP, the producer does not
> > > >
> > > > rely
> > > >
> > > > on
> > > >
> > > > the broker to autocreate--if the broker's config is disabled, then
> > > >
> > > > the
> > > >
> > > > producer can autocreate on its own with a create topic request (the
> > > >
> > > > same
> > > >
> > > > type of request the admin client uses).
> > > > However, if both configs are enabled, the broker will autocreate
> > > >
> > > > through
> > > >
> > > > a
> > > >
> > > > metadata request before the producer gets a chance. Of course, the
> > > >
> > > > way
> > > >
> > > > to
> > > >
> > > > avoid this, is to do as you suggested, and set
> > > >
> > > > the
> > > >
> > > > "allow_auto_topic_creation" field to false.
> > > >
> > > > I think the only thing we need to be careful with in this setup is
> > > >
> > > > without
> > > >
> > > > KIP 464, we can not use broker defaults for this topic. A user
> > > >
> > > > needs
> > > >
> > > > to
> > > >
> > > > specify the number of partition and replication factor in the
> > > >
> > > > config.
> > > >
> > > > An
> > > >
> > > > alternative to this is to have coded defaults for when these
> > > >
> > > > configs
> > > >
> > > > are
> > > >
> > > > unspecified, but it is not immediately apparent what these defaults
> > > >
> > > > should
> > > >
> > > > be.
> > > >
> > > > Thanks again for reading my KIP,
> > > > Justine
> > > >
> > > > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> > > >
> > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the response!
> > > > In my opinion, it would be better if the producer did not rely at
> > > >
> > > > all
> > > >
> > > > on the broker auto create feature as this is what we're aiming to
> > > > deprecate. When requesting metadata we can set the
> > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > creation. Then if the topic is not existing, send a
> > > >
> > > > CreateTopicRequest.
> > > >
> > > > What do you think?
> > > >
> > > > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Currently the way it is implemented, the broker auto-creation
> > > >
> > > > configuration
> > > >
> > > > takes precedence. The producer will not use the CreateTopics
> > > >
> > > > request.
> > > >
> > > > (Technically it can--but the topic will already be created
> > > >
> > > > through
> > > >
> > > > the
> > > >
> > > > broker, so it will never try to create the topic.) It is possible
> > > >
> > > > to
> > > >
> > > > change this however, and I'd be happy to
> > > >
> > > > discuss
> > > >
> > > > the
> > > >
> > > > benefits of this alternative.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> > > >
> > > > mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > In case auto creation is enabled on both the client and server,
> > > >
> > > > will
> > > >
> > > > the producer still use the AdminClient (CreateTopics request)
> > > >
> > > > to
> > > >
> > > > create topics? and not rely on the broker auto create. I'm
> > > >
> > > > guessing the
> > > >
> > > > answer is yes but can you make it explicit.
> > > >
> > > > Thank you
> > > >
> > > > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi,
> > > > Just a friendly reminder to take a look at this KIP if you
> > > >
> > > > have
> > > >
> > > > the
> > > >
> > > > time.
> > > >
> > > > I was thinking about broker vs. client default precedence,
> > > >
> > > > and I
> > > >
> > > > think it
> > > >
> > > > makes sense to keep the broker as the default used when both
> > > >
> > > > client-side
> > > >
> > > > and broker-side defaults are configured. The idea is that
> > > >
> > > > there
> > > >
> > > > would be
> > > >
> > > > pretty clear documentation, and that many systems with
> > > >
> > > > configurations
> > > >
> > > > that
> > > >
> > > > the client could not change would likely have the auto-create
> > > >
> > > > default
> > > >
> > > > off.
> > > >
> > > > (In cloud for example).
> > > >
> > > > It also seems like in most cases, the consumer config
> > > > ' allow. auto. create. topics ( http://allow.auto.create.topics/ ) '
> > was
> > > > created to actually prevent
> > > >
> > > > the
> > > >
> > > > creation
> > > >
> > > > of
> > > >
> > > > topics, so the loss of creation functionality will not be a
> > > >
> > > > big
> > > >
> > > > problem.
> > > >
> > > > I'm happy to discuss any other compatibility problems or
> > > >
> > > > components
> > > >
> > > > of
> > > >
> > > > this KIP.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io )
> > > >
> > > > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I was looking at this KIP again, and there is a decision I
> > > >
> > > > made
> > > >
> > > > that I
> > > >
> > > > think is worth discussing.
> > > >
> > > > In the case where both the broker and producer's
> > > > 'auto.create.topics.enable' are set to true, we have to
> > > >
> > > > choose
> > > >
> > > > either
> > > >
> > > > the
> > > >
> > > > broker configs or the producer configs for the replication
> > > > factor/partitions.
> > > >
> > > > Currently, the decision is to have the broker defaults take
> > > >
> > > > precedence.
> > > >
> > > > (It is easier to do this in the implementation.) It also
> > > >
> > > > makes
> > > >
> > > > some
> > > >
> > > > sense
> > > >
> > > > for this behavior to take precedence since this behavior
> > > >
> > > > already
> > > >
> > > > occurs as
> > > >
> > > > the default.
> > > >
> > > > However, I was wondering if it would be odd for those who
> > > >
> > > > can
> > > >
> > > > only
> > > >
> > > > see
> > > >
> > > > the
> > > >
> > > > producer side to set configs for replication
> > > >
> > > > factor/partitions
> > > >
> > > > and
> > > >
> > > > see
> > > >
> > > > different results. Currently the documentation for the
> > > >
> > > > config
> > > >
> > > > states
> > > >
> > > > that
> > > >
> > > > the config values are only used when the broker config is
> > > >
> > > > not
> > > >
> > > > enabled,
> > > >
> > > > but
> > > >
> > > > this might not always be clear to the user. Changing the
> > > >
> > > > code
> > > >
> > > > to
> > > >
> > > > have
> > > >
> > > > the
> > > >
> > > > producer's configurations take precedence is possible, but
> > > >
> > > > I
> > > >
> > > > was
> > > >
> > > > wondering
> > > >
> > > > what everyone thought.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Just a quick update--
> > > >
> > > > It seems that enabling both the broker and producer
> > > >
> > > > configs
> > > >
> > > > works
> > > >
> > > > fine,
> > > >
> > > > except that the broker configurations for partitions,
> > > >
> > > > replication
> > > >
> > > > factor
> > > >
> > > > take precedence.
> > > > I don't know if that is something we would want to
> > > >
> > > > change, but
> > > >
> > > > I'll be
> > > >
> > > > updating the KIP for now to reflect this. Perhaps we would
> > > >
> > > > want to
> > > >
> > > > add more
> > > >
> > > > to the documentation of the the producer configs to
> > > >
> > > > clarify.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Colin,
> > > >
> > > > Thanks for looking at the KIP. I can definitely add to
> > > >
> > > > the
> > > >
> > > > title
> > > >
> > > > to
> > > >
> > > > make
> > > >
> > > > it more clear.
> > > >
> > > > It makes sense that both configurations could be turned
> > > >
> > > > on
> > > >
> > > > since
> > > >
> > > > there
> > > >
> > > > are many cases where the user can not control the
> > > >
> > > > server-side
> > > >
> > > > configurations. I was a little concerned about how both
> > > >
> > > > interacting
> > > >
> > > > would
> > > >
> > > > work out -- if there would be to many requests for new
> > > >
> > > > topics,
> > > >
> > > > for
> > > >
> > > > example.
> > > >
> > > > But it since it does make sense to allow both
> > > >
> > > > configurations
> > > >
> > > > enabled, I
> > > >
> > > > will test out some scenarios and I'll change the KIP to
> > > >
> > > > support
> > > >
> > > > this.
> > > >
> > > > I also agree with documentation about distinguishing the
> > > >
> > > > differences. I
> > > >
> > > > was having some trouble with the wording but I like the
> > > >
> > > > phrases
> > > >
> > > > "server-side" and "client-side." That's a good
> > > >
> > > > distinction I
> > > >
> > > > can
> > > >
> > > > use
> > > >
> > > > when
> > > >
> > > > describing.
> > > >
> > > > I'll try to update the KIP soon keeping everyone's input
> > > >
> > > > in
> > > >
> > > > mind.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> > > >
> > > > cmccabe@ apache. org ( cmccabe@apache.org )
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP. This seems like a good step towards
> > > >
> > > > removing
> > > >
> > > > server-side topic auto-creation.
> > > >
> > > > We should add included "client-side" to the title of
> > > >
> > > > the KIP
> > > >
> > > > somewhere,
> > > >
> > > > to make it clear that we're talking about client-side
> > > >
> > > > auto
> > > >
> > > > creation.
> > > >
> > > > The KIP says:
> > > >
> > > > In order to automatically create topics with the
> > > >
> > > > producer, the
> > > >
> > > > producer's
> > > >
> > > > auto.create.topics.enable config must be set to true
> > > >
> > > > and
> > > >
> > > > the
> > > >
> > > > broker
> > > >
> > > > config should be set to false
> > > >
> > > > From a user's point of view, this seems
> > > >
> > > > counter-intuitive.
> > > >
> > > > In
> > > >
> > > > order to
> > > >
> > > > auto-create topics the broker's
> > > >
> > > > auto.create.topics.enable
> > > >
> > > > config
> > > >
> > > > should be
> > > >
> > > > set to false? It seems like the server-side
> > > >
> > > > auto-create is
> > > >
> > > > unrelated to
> > > >
> > > > the client-side auto-create. We could have both turned
> > > >
> > > > on
> > > >
> > > > (and
> > > >
> > > > I'm
> > > >
> > > > sure
> > > >
> > > > that in the real world, people will try this
> > > >
> > > > configuration...)
> > > >
> > > > There's no
> > > >
> > > > reason not to support this, I think.
> > > >
> > > > We should add some documentation explaining the
> > > >
> > > > difference
> > > >
> > > > between
> > > >
> > > > server-side and client-side auto-creation. Without
> > > >
> > > > documentation,
> > > >
> > > > an admin
> > > >
> > > > might think that they had disabled all forms of
> > > >
> > > > auto-creation by
> > > >
> > > > setting
> > > >
> > > > the -side setting to false-- but this is not the case,
> > > >
> > > > of
> > > >
> > > > course.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > >
> > > > Hi Dhruvil,
> > > >
> > > > Thanks for reading the KIP!
> > > > That was the general idea for deprecation. We would
> > > >
> > > > log a
> > > >
> > > > warning
> > > >
> > > > when the
> > > >
> > > > config is enabled on the broker.
> > > > I also believe that there would be a change to
> > > >
> > > > documentation.
> > > >
> > > > If there is anything else that should be done, please
> > > >
> > > > let
> > > >
> > > > me
> > > >
> > > > know!
> > > >
> > > > Justine
> > > >
> > > > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> > > >
> > > > dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP, this is great!
> > > >
> > > > Could you add some more information about what
> > > >
> > > > deprecating
> > > >
> > > > the
> > > >
> > > > broker
> > > >
> > > > configuration means? Would we log a warning in the
> > > >
> > > > logs
> > > >
> > > > when
> > > >
> > > > auto
> > > >
> > > > topic
> > > >
> > > > creation is enabled on the broker, for example?
> > > >
> > > > Thanks,
> > > > Dhruvil
> > > >
> > > > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> > > >
> > > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > > >
> > > > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I'd like to start a discussion thread for KIP-487. This KIP plans
> > > >
> > > > to
> > > >
> > > > deprecate the current system of
> > > >
> > > > auto-creating
> > > >
> > > > topics
> > > >
> > > > through requests to the metadata and give the
> > > >
> > > > producer the
> > > >
> > > > ability to
> > > >
> > > > automatically create topics instead.
> > > >
> > > > More information can be found here:
> > > >
> > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > > https://cwiki.
> > > > apache.org/confluence/display/KAFKA/ )
> > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > >
> > > > Thank you,
> > > > Justine Olshan
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by "M. Manna" <ma...@gmail.com>.

Hi,

If I may, perhaps you could simplify everything by using only
'auto.create.topics.enable' as a value along with true. In other words, the
public interfaces section should only have [true,auto.create.topics.enable,
false].

The reason for this is that auto.create.topics.enable is already known to
users as a "Server-SIde" config. So all you are saying is

a) To avoid day 1 impact, it will follow whatever auto.create.topics.enable
value is set.
b) False means - no client side topic creation
c) True means client side topic creation.

It saves creating 2 more new strings :). But not too expensive anyway.

Also, when you deprecate auto.create.topics.enable - you must provide
sufficient logic to ensure that things like rolling upgrade doesn't
temporarily break anything. I apologise if you have already accounted for
this, but wanted to mention since I didn't notice this on the KIP.

Let me know how this sounds.

Regards,

On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jo...@confluent.io> wrote:

> Hi Harsha,
>
> I think my message may have gotten lost in all the others.
>
> Two of the goals of this KIP are to 1) allow auto-creation on specific
> clients when the broker default is false and 2) eventually replace the
> broker config.
>
> In order to accomplish these two goals, we need the producer to be able to
> create topics despite the broker config. (How can we replace a function
> when we rely on it?)
> I think at this point we have a fundamental disagreement in what we should
> allow the producer to do.
> In my previous message I mentioned a config that would allow for the broker
> to prevent producer auto-creation. (It would be disabled by default.) It
> would fix your issue for now, but could lead to more complications later.
>
> Thank you,
> Justine
>
>
> On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz> wrote:
> >
> > > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> > >
> > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmccabe@apache.org >
> > > wrote:
> > >
> > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> > >
> > > Hi Colin,
> > > "Hmm... I'm not sure I follow. Users don't have to build their own
> > > tooling, right? They can use any of the shell scripts that we've
> shipped
> > in
> > > the last few releases. For example, if any of your users run it, this
> > shell
> > > script will delete all of the topics from your non-security-enabled
> > > cluster:
> > >
> > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --delete
> > > --topic
> > >
> > > They will need to fill in the correct bootstrap servers list, of
> course,
> > > not localhost. This deletion script will work on some pretty old
> brokers,
> > > even back to the 0.10 releases. It seems a little odd to trust your
> users
> > > with this power, but not trust them to avoid changing a particular
> > > configuration key."
> > >
> > > The above will blocked by the server if we set delete.topic.enable to
> > > false and thats exactly what I am asking for.
> > >
> > > Hi Harsha,
> > >
> > > I was wondering if someone was going to bring up that configuration :)
> > >
> > > it's an interesting complication, but globally disabling topic deletion
> > is
> > > not very practical for most use-cases.
> > >
> > > In any case, there are plenty of other bad things that users with full
> > > permissions can do that aren't blocked by any server configuration. For
> > > example, they can delete every record in every topic. I can write a
> > script
> > > for that too, and there's no server configuration you can set to
> disable
> > > it. Or I could simply create hundreds of thousands of topics, until
> > cluster
> > > performance becomes unacceptable (this will be even more of a problem
> if
> > > someone configured delete.topic.enable as false). Or publish bad data
> to
> > > every topic, etc. etc.
> > >
> > > The point I'm trying to make here is that you can't rely on these kind
> of
> > > server-side configurations for security. At most, they're a way to set
> up
> > > certain very simple policies. But the policies are so simple that
> they're
> > > hardly ever useful any more.
> > >
> > > For example, if the problem you want to solve is that you want a user
> to
> > > only be able to create 50 topics and not delete anyone else's topics,
> you
> > > can solve that with a CreateTopicsPolicy that limits the number of
> > topics,
> > > and some ACLs. There's no combination of auto.create.topics.enable and
> > > delete.topic.enable that will help here.
> > >
> > > Hi Colin,
> > >
> > >             Well you gave the example that a user can delete the topics
> > > just by running that script  :).
> > >
> > > I understand there are open APIs in Kafka and can lead to rogue clients
> > > taking advantage of it without proper security in place.
> > >
> > > What I am asking so far in this thread is , this KIP is changing the
> > > producer behavior and its not backward compatible.
> > >
> > > The change is backwards compatible. The default will still be
> server-side
> > > topic auto-creation, just like now.
> > >
> > You will have to specifically change the producer config to get the new
> > > behavior.
> > >
> >
> >
> > I disagree.  Today server can turn off the topic creation  neither
> producer
> > or consumer can create a topic. With this KIP , producer can create a
> topic
> > by turning on client side config when server side config is turned off.
> >
> >
> > We can still achieve
> > > the main goal of the KIP which is to change MetadataRequest  creating
> > > topics and send a CreateTopicRequest from Producer and also keep the
> > server
> > > side config to have precedence.  This KIP originally written to have
> > server
> > > side preference and there is not much explanation why it changed to
> have
> > > Producer side preference.
> > >
> > > Arguing that AdminClient can do that and so we are going to make
> Producer
> > > do the same doesn't make sense.
> > >
> > > "The downside is that if we wanted to check a server side configuration
> > > before sending the create topics request, the code would be more
> complex.
> > > The behavior would also not be consistent with how topic auto-creation
> is
> > > handled in Kafka Streams."
> > >
> > > I am not sure why you need to check server side configuration before
> > > sending create topics request. A user enables producer side config to
> > > create topics.
> > > Producer sends a request to the broker and if the broker has
> > > auto.topic.create.enable to true (default) it will allow creation of
> > > topics. If it set to false it returns error back to the client.
> > >
> > > auto.topic.create.enable has never affected CreateTopicsRequest. If you
> > > submit a CreateTopicsRequest and you are authorized, a topic will be
> > > created, regardless of what the value of auto.topic.create.enable is.
> > This
> > > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?)
> > >
> > > I don't see how this behavior will be different in Kafka streams. By
> > > default server allows the topic creation and with this KIP, It will
> only
> > > allow creation of topic when both producer and server side are turned
> on.
> > > Its exactly the same behavior in KIP-361.
> > >
> > > I suggest running a streams job as a test. It creates the topics it
> needs
> > > using AdminClient. The value of auto.topic.create.enable is not
> relevant.
> > > Whether it is true or false, the topics will still be created.
> > >
> > > "In general, it would be nice if we could keep the security and access
> > > control model simple and not introduce a lot of special cases and
> > > exceptions. Kafka has basically converged on using ACLs and
> > > CreateTopicPolicy classes to control who can create topics and where.
> > > Adding more knobs seems like a step backwards, especially when the
> > proposed
> > > knobs don't work consistently across components, and don't provide true
> > > security." This is not about access control at all. Shipping sane
> > defaults
> > > should be prioritized.
> > >
> > > I don't think the default is really the question here. I think everyone
> > > agrees that the default for client-side auto-topic creation should be
> > off.
> > >
> > > My point goes back to KIP-361 why we kept the server side config to
> have
> > > preference. We can keep bringing back the discussion of security
> policies
> > > but a producer client overiding server side setting is not just
> security
> > > policy issue .
> > >
> > > "Producer client can override server side setting and not consumer and
> > > also producer can only override creation of topic but no the
> replication
> > > and partitions. " This doesn't make sense to me and why we want to
> > > introduce such a confusing behavior.
> > >
> > > The scenarios that you're describing (such as dealing with a poorly
> > > configured client that tries to create topics it should not) do seem to
> > be
> > > about access control.
> > >
> > > We keep talking about CreateTopicPolicy and yet we don't have default
> one
> > > and asking every user of Kafka implement their own doesn't make sense
> > here.
> > >
> > > The point of CreateTopicPolicy is exactly that you can implement your
> > own,
> > > though. It's a way of customizing what the policy is in your specific
> > > cluster.
> > >
> > > I agree that most people don't want to write a custom policy. But most
> of
> > > those people are satisfied with just setting up ACLs to set up simple
> > > policies like this user can create topics, that user can't, etc. That's
> > why
> > > I suggested adding support for ACLs to insecure clusters might be
> > helpful.
> > >
> > > Also, just as a side note, wouldn't it would be impossible for us to
> > > specify a default CreateTopicsPolicy that did anything at this point
> > anyway
> > > without breaking backwards compatibility? Maybe I'm misunderstanding
> the
> > > proposal.
> > >
> > > I am asking about exact behavior that we shipped for consumer side
> > https:/
> > > / cwiki. apache. org/ confluence/ display/ KAFKA/
> > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > (
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > > )
> > >
> > > I still didn't get any response why this behavior shouldn't be exactly
> > > like Kafka consumer and why do we want to have producer to overrider
> > server
> > > side config and while not allowing consumer to do so. We are not even
> > > allowing the same contract and this will cause more confusion from the
> > > users standpoint.
> > >
> > > Hmm. The consumer already has a way to override the server side
> > > configuration. See KIP-361: Add Consumer Configuration to Disable Auto
> > > Topic Creation. This lets the consumer skip auto-creating topics, even
> if
> > > auto-creation is enabled on the broker.
> > >
> > > Yes that's what I am saying and it doesn't allow topic creation if the
> > > server side auto-creation is disabled and in consumer its enabled.   I
> > > would like to see the exact behavior for producer as stated in KIP-361.
> > >
> > > To be fair, the KIP should probably discuss why we don't want to
> > implement
> > > client-side topic creation in the consumer in "rejected alternatives."
> > > Maybe Justine can add more context here in the KIP.
> > >
> > > The last time we talked about this, the reasoning was that we wanted to
> > > eventually get rid of consumer-side auto-topic creation entirely, and
> so
> > it
> > > wasn't worth implementing an improved way of doing it. Producer side
> auto
> > > topic-creation, in contrast, will probably stick around, although we'd
> > like
> > > to deprecate and remove the problematic server-side mechanism over
> time.
> > >
> > > We should implement the producer side topic creation with proper create
> > > topic request  and I am in favor of this KIP.
> > >
> > > All I am asking is don't make the producer to override server side
> config
> > > and keep it similar to KIP-361 just like consumer, it honors what set
> on
> > > server side which takes the precedence.   Changing this behavior in
> > > producer is backward incompatible and will catch users by surprise.
> > >
> > > All arguments for enforcing producer side overriding config can still
> be
> > > achieved. Server side auto creation turned on by default and  any new
> > > producer client setting their auto creation config to true will create
> > > topics in default behavior and any user who set server side to false
> and
> > > upgrades to latest kafka with this KIP will not see any unwanted
> behavior
> > > from clients.  IMO this is not a unreasonable request on this KIP and
> > this
> > > requested behavior is exactly what the KIP initially proposed.
> > >
> > > Thanks,
> > >
> > > Harsha
> > >
> > > best,
> > > Colin
> > >
> > > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache. org (
> > > cmccabe@apache.org ) > wrote:
> > >
> > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > >
> > > Not sure how the AdminClient applies here, It is an external API and is
> > > not part of KafkaProducer so any user who updates to latest version of
> > > Kafka with this feature get to create the topics.
> > > They have to build a tooling around AdminClient allow themselves to
> > >
> > > create
> > >
> > > topics.
> > >
> > > Hi Harsha,
> > >
> > > Hmm... I'm not sure I follow. Users don't have to build their own
> > tooling,
> > > right? They can use any of the shell scripts that we've shipped in the
> > last
> > > few releases. For example, if any of your users run it, this shell
> script
> > > will delete all of the topics from your non-security-enabled cluster:
> > >
> > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --delete
> > > --topic
> > >
> > > They will need to fill in the correct bootstrap servers list, of
> course,
> > > not localhost. This deletion script will work on some pretty old
> brokers,
> > > even back to the 0.10 releases. It seems a little odd to trust your
> users
> > > with this power, but not trust them to avoid changing a particular
> > > configuration key.
> > >
> > > There is no behavior in Kafka producer that allowed users to delete the
> > > topics or delete the records. So citing them as an example doesn't
> makes
> > > sense in this context.
> > >
> > > I think Kafka Streams is relevant here. After all, it's software that
> we
> > > ship as part of the official Kafka release. And it auto-creates topics
> > even
> > > when auto.create.topics.enable is set to false on the broker.
> > >
> > > I think that auto.create.topics.enable was never intended as a security
> > > setting to control access. It was intended as a way to disable the
> > > broker-side auto-create feature specifically, because there were some
> > > problems with that specific feature. Broker-side auto-creation is
> > > frustrating because it's on by default, and because it can auto-create
> > > topics even if the producer or consumer didn't explicitly ask for that.
> > > Neither of those problems applies to this KIP: producers have to
> > > specifically opt in, and it won't be on by default. Basically, we think
> > > that client-side auto-creation is actually a lot better-- hence this
> KIP
> > > :)
> > >
> > > But there is
> > > a functionality which allowed creation of topics if they don't exist in
> > >
> > > the
> > >
> > > cluster and this behavior could be controlled by a config on the server
> > > side. Now with this KIP we are allowing producer to make that decision
> > > without any gateway on the server via configs. Any user who is not
> aware
> > >
> > > of
> > >
> > > such changes
> > > can accidentally create these topics and we are essentially removing a
> > > config that exists in brokers today to block this accidental creation
> and
> > > allowing clients to take control.
> > >
> > > Again, I hope I'm not misinterpreting, but I don't see how this can be
> > > turned on accidentally. The user would have to specifically turn this
> on
> > in
> > > the producer by setting the configuration key.
> > >
> > > I still didn't get any positives of not having server side configs? if
> > you
> > > want to turn them on and allow any client to create topics set the
> > default
> > > to true
> > > and allow users who doesn't want to have this feature let them turn
> them
> > > off. It will be the exact behavior as it is today , as far as producer
> is
> > > concerned. I am not
> > > understanding why not having server side configs to gateways such a
> hard
> > > requirement and this behavior exists today. As far I am concerned this
> > > breaks the backward compatibility.
> > >
> > > The downside is that if we wanted to check a server side configuration
> > > before sending the create topics request, the code would be more
> complex.
> > > The behavior would also not be consistent with how topic auto-creation
> is
> > > handled in Kafka Streams.
> > >
> > > In general, it would be nice if we could keep the security and access
> > > control model simple and not introduce a lot of special cases and
> > > exceptions. Kafka has basically converged on using ACLs and
> > > CreateTopicPolicy classes to control who can create topics and where.
> > > Adding more knobs seems like a step backwards, especially when the
> > proposed
> > > knobs don't work consistently across components, and don't provide true
> > > security.
> > >
> > > Maybe we should support an insecure mode where there are still users
> and
> > > ACLs. That would help people who don't want to set up Kerberos, etc.
> but
> > > still want to protect against misconfigured clients or accidents.
> Hadoop
> > > has something like this, and I think it was useful. NFS also supported
> > > (supports?) a mode where you just pass whatever user ID you want and
> the
> > > system believes you. These things clearly don't protect against
> malicious
> > > users, but they can help set up policies when needed.
> > >
> > > best,
> > > Colin
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache. org (
> > > cmccabe@apache.org ) > wrote:
> > >
> > > Hi Harsha,
> > >
> > > Thanks for taking a look.
> > >
> > > I'm not sure I follow the discussion about AdminClient.
> > >
> > > KafkaAdminClient
> > >
> > > has been around for about 2 years at this point as a public class.
> > >
> > > There
> > >
> > > are many programs that use it to automatically create topics. Kafka
> > > Streams does this, for example. If any of your users run Kafka
> > >
> > > Streams,
> > >
> > > they will be auto-creating topics, regardless of what setting you use
> > >
> > > for
> > >
> > > auto.create.topics.enable.
> > >
> > > A big part of the annoyance of auto-topic creation right now is that
> > >
> > > it is
> > >
> > > on by default. The new configuration proposed by KIP-487 wouldn't be.
> > > Users would have to explicitly opt in to the new behavior of
> > >
> > > client-side
> > >
> > > topic creation. If you run without security, you're already putting a
> > >
> > > huge
> > >
> > > amount of trust in your users. For example, you trust them not to
> > >
> > > delete
> > >
> > > records with the kafka-delete-records. sh (
> http://kafka-delete-records.
> > > sh/
> > > ) command, or delete topics with kafka-topics. sh (
> http://kafka-topics.
> > > sh/
> > > ). Trusting them not to set a certain config value seems minor in
> > > comparison, right?
> > >
> > > best,
> > > Colin
> > >
> > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > >
> > > Hi,
> > > Even with policies one needs to implement that, so for every
> > >
> > > user who
> > >
> > > doesn't want a producer to create topics or have limits around
> > >
> > > partitions &
> > >
> > > replication factor they have to implement a policy. The KIP is changing
> > > the behavior , it might not be introducing
> > >
> > > the
> > >
> > > new functionality but it will enable producers to override the create
> > >
> > > topic
> > >
> > > config settings on the broker. What I am asking for to provide a
> > >
> > > config
> > >
> > > that will disable auto creation of topics and if its enabled set some
> > >
> > > sane
> > >
> > > defaults so that clients can create a topic with in those limits. I
> > >
> > > don't
> > >
> > > see how this not related to this KIP.
> > > If the server config options as I suggested doesn't interest
> > >
> > > you at
> > >
> > > least have a default CreateTopicPolices in place.
> > > To give an example, In our environment we disable the
> > > auto.create.topic.enable and force users to go through a centralized
> > > service as we want capture more details about the topic creation and
> > > requirements. With this KIP, a producer can create a topic with no
> > >
> > > bounds.
> > >
> > > Another example max.message.size we define that at cluster level
> > >
> > > and one
> > >
> > > can override max.messsage.size at topic level, any misconfiguration
> > >
> > > at
> > >
> > > this
> > >
> > > will cause service degradation. Its not always about the rogue
> > >
> > > clients,
> > >
> > > users can easily misconfigure and can cause an outage. Again we can
> talk
> > > about CreateTopicPolicy but without having a
> > >
> > > default
> > >
> > > implementation and asking everyone to implement their own while
> > >
> > > changing
> > >
> > > the behavior in producer doesn't make sense to me.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me. uk (
> > > ismael@juma.me.uk ) >
> > >
> > > wrote:
> > >
> > > Hi Harsha,
> > >
> > > I mentioned policies and the authorizer. For example, with
> > > CreateTopicPolicy, you can implement the limits you describe. If
> > >
> > > you
> > >
> > > have
> > >
> > > ideas of how that should be improved, please submit a KIP. My
> > >
> > > point is
> > >
> > > that
> > >
> > > this KIP is not introducing any new functionality with regards to
> > >
> > > what
> > >
> > > rogue clients can do. It's using the existing protocol that is
> > >
> > > already
> > >
> > > exposed via the AdminClient. So, I don't think we need to address
> > >
> > > it in
> > >
> > > this KIP. Does that make sense?
> > >
> > > Ismael
> > >
> > > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> > >
> > > kafka@ harsha. io ( kafka@harsha.io ) >
> > >
> > > wrote:
> > >
> > > Ismael,
> > > Sure AdminClient can do that and we should've shipped a config or
> > >
> > > use
> > >
> > > the
> > >
> > > existing one to block that. Not all users are yet to upgrade to
> > >
> > > AdminClient
> > >
> > > and start using that to cause issues yet. In shared environment we
> > >
> > > should
> > >
> > > allow server to set sane defaults and not allow every client to go
> > >
> > > ahead
> > >
> > > create random no.of topic/partitions and replication factor. Even
> > >
> > > if
> > >
> > > the
> > >
> > > users want to allow topic creation proposed in the KIP , it makes
> > >
> > > sense to
> > >
> > > have some guards against the no.of partitions and replication
> > >
> > > factor.
> > >
> > > Authorizer is not always an answer to block requests and having
> > >
> > > users
> > >
> > > set
> > >
> > > server side configs to protect a multi-tenant environment is
> > >
> > > required.
> > >
> > > In a
> > >
> > > non-secure environment Authorizer is a blunt instrument either you
> > >
> > > end
> > >
> > > up
> > >
> > > blocking everyone or allowing everyone.
> > > I am asking to have server side that allow clients to create
> > >
> > > topics or
> > >
> > > not
> > >
> > > , if they are allowed set a ceiling on max no.of partitions and
> > > replication-factor.
> > >
> > > -Harsha
> > >
> > > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com (
> ismaelj@gmail.com
> > > )
> > >
> > > wrote:
> > >
> > > Harsha,
> > >
> > > Rogue clients can use the admin client to create topics and
> > >
> > > partitions.
> > >
> > > ACLs and policies can help in that case as well as this one.
> > >
> > > Ismael
> > >
> > > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha. io
> (
> > > kafka@harsha.io ) >
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > > Thanks for the KIP.
> > > "When server-side auto-creation is disabled, client-side
> > >
> > > auto-creation
> > >
> > > will try to use client-side configurations"
> > > If I understand correctly, this KIP is removing any server-side
> > >
> > > blocking
> > >
> > > client auto creation of topic?
> > > if so this will present potential issue of rogue client creating
> > >
> > > ton of
> > >
> > > topic-partitions and potentially bringing down the service for
> > >
> > > everyone
> > >
> > > or
> > >
> > > degrade the service itself.
> > > By reading the KIP its not clear to me that there is a clear way to
> > >
> > > block
> > >
> > > auto creation topics of all together from clients by server side
> > >
> > > config.
> > >
> > > Server side configs of default topic, partitions should take higher
> > > precedence and client shouldn't be able to create a topic with
> > >
> > > higher
> > >
> > > no.of
> > >
> > > partitions, replication than what server config specifies.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hi all,
> > > I made some changes to the KIP. Hopefully this configuration change
> > >
> > > will
> > >
> > > make things a little clearer.
> > >
> > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > https://cwiki.
> > > apache.org/confluence/display/KAFKA/ )
> > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > >
> > > Please let me know if you have any feedback or questions!
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache. org (
> > > cmccabe@apache.org ) >
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think you bring up a good point. It would be better if we didn't
> > >
> > > ever
> > >
> > > have to set up client-side configuration for this feature, and
> > >
> > > KIP-464
> > >
> > > would let us skip this entirely.
> > >
> > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > >
> > > Hi Mickael,
> > > I agree that KIP-464 works on newer brokers, but I was a bit
> > >
> > > worried
> > >
> > > how
> > >
> > > things would play out on older brokers that* do not *have KIP 464
> > >
> > > included.
> > >
> > > Is it enough to throw an error in this case when producer configs
> > >
> > > are
> > >
> > > not
> > >
> > > specified?
> > >
> > > I think the right thing to do would be to log an error message in
> > >
> > > the
> > >
> > > client. We will need to have that capability in any case, to cover
> > > scenarios like the client trying to auto-create a topic that they
> > >
> > > don't
> > >
> > > have permission to create. Or a client trying to create a topic on
> > >
> > > a
> > >
> > > broker
> > >
> > > so old that CreateTopicsRequest is not supported.
> > >
> > > The big downside to relying on KIP-464 is that it is a very recent
> > >
> > > feature
> > >
> > > -- so recent that it hasn't even made its way to any official
> > >
> > > Apache
> > >
> > > release. It's scheduled for the upcoming 2.4 release in a few
> > >
> > > months.
> > >
> > > So if you view this KIP as a step towards removing broker-side
> > > auto-create, you might want to support older brokers just to
> > >
> > > accelerate
> > >
> > > adoption, and hasten the day when we can finally flip broker-side
> > > auto-create to off (or even remove it entirely).
> > >
> > > I have to agree, though, that having client-side configurations for
> > >
> > > number
> > >
> > > of partitions and replication factor is messy. Maybe it would be
> > >
> > > worth
> > >
> > > it
> > >
> > > to restrict support to post-KIP-464 brokers, if we could avoid
> > >
> > > creating
> > >
> > > more configs.
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> > >
> > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > We can rely on KIP-464 which allows to omit the partition count or
> > > replication factor when creating a topic. In that case, the broker
> > >
> > > defaults
> > >
> > > are used.
> > >
> > > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Michael,
> > > That makes sense to me!
> > > To clarify, in the current state of the KIP, the producer does not
> > >
> > > rely
> > >
> > > on
> > >
> > > the broker to autocreate--if the broker's config is disabled, then
> > >
> > > the
> > >
> > > producer can autocreate on its own with a create topic request (the
> > >
> > > same
> > >
> > > type of request the admin client uses).
> > > However, if both configs are enabled, the broker will autocreate
> > >
> > > through
> > >
> > > a
> > >
> > > metadata request before the producer gets a chance. Of course, the
> > >
> > > way
> > >
> > > to
> > >
> > > avoid this, is to do as you suggested, and set
> > >
> > > the
> > >
> > > "allow_auto_topic_creation" field to false.
> > >
> > > I think the only thing we need to be careful with in this setup is
> > >
> > > without
> > >
> > > KIP 464, we can not use broker defaults for this topic. A user
> > >
> > > needs
> > >
> > > to
> > >
> > > specify the number of partition and replication factor in the
> > >
> > > config.
> > >
> > > An
> > >
> > > alternative to this is to have coded defaults for when these
> > >
> > > configs
> > >
> > > are
> > >
> > > unspecified, but it is not immediately apparent what these defaults
> > >
> > > should
> > >
> > > be.
> > >
> > > Thanks again for reading my KIP,
> > > Justine
> > >
> > > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> > >
> > > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the response!
> > > In my opinion, it would be better if the producer did not rely at
> > >
> > > all
> > >
> > > on the broker auto create feature as this is what we're aiming to
> > > deprecate. When requesting metadata we can set the
> > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > creation. Then if the topic is not existing, send a
> > >
> > > CreateTopicRequest.
> > >
> > > What do you think?
> > >
> > > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Currently the way it is implemented, the broker auto-creation
> > >
> > > configuration
> > >
> > > takes precedence. The producer will not use the CreateTopics
> > >
> > > request.
> > >
> > > (Technically it can--but the topic will already be created
> > >
> > > through
> > >
> > > the
> > >
> > > broker, so it will never try to create the topic.) It is possible
> > >
> > > to
> > >
> > > change this however, and I'd be happy to
> > >
> > > discuss
> > >
> > > the
> > >
> > > benefits of this alternative.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> > >
> > > mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP!
> > >
> > > In case auto creation is enabled on both the client and server,
> > >
> > > will
> > >
> > > the producer still use the AdminClient (CreateTopics request)
> > >
> > > to
> > >
> > > create topics? and not rely on the broker auto create. I'm
> > >
> > > guessing the
> > >
> > > answer is yes but can you make it explicit.
> > >
> > > Thank you
> > >
> > > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hi,
> > > Just a friendly reminder to take a look at this KIP if you
> > >
> > > have
> > >
> > > the
> > >
> > > time.
> > >
> > > I was thinking about broker vs. client default precedence,
> > >
> > > and I
> > >
> > > think it
> > >
> > > makes sense to keep the broker as the default used when both
> > >
> > > client-side
> > >
> > > and broker-side defaults are configured. The idea is that
> > >
> > > there
> > >
> > > would be
> > >
> > > pretty clear documentation, and that many systems with
> > >
> > > configurations
> > >
> > > that
> > >
> > > the client could not change would likely have the auto-create
> > >
> > > default
> > >
> > > off.
> > >
> > > (In cloud for example).
> > >
> > > It also seems like in most cases, the consumer config
> > > ' allow. auto. create. topics ( http://allow.auto.create.topics/ ) '
> was
> > > created to actually prevent
> > >
> > > the
> > >
> > > creation
> > >
> > > of
> > >
> > > topics, so the loss of creation functionality will not be a
> > >
> > > big
> > >
> > > problem.
> > >
> > > I'm happy to discuss any other compatibility problems or
> > >
> > > components
> > >
> > > of
> > >
> > > this KIP.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io )
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I was looking at this KIP again, and there is a decision I
> > >
> > > made
> > >
> > > that I
> > >
> > > think is worth discussing.
> > >
> > > In the case where both the broker and producer's
> > > 'auto.create.topics.enable' are set to true, we have to
> > >
> > > choose
> > >
> > > either
> > >
> > > the
> > >
> > > broker configs or the producer configs for the replication
> > > factor/partitions.
> > >
> > > Currently, the decision is to have the broker defaults take
> > >
> > > precedence.
> > >
> > > (It is easier to do this in the implementation.) It also
> > >
> > > makes
> > >
> > > some
> > >
> > > sense
> > >
> > > for this behavior to take precedence since this behavior
> > >
> > > already
> > >
> > > occurs as
> > >
> > > the default.
> > >
> > > However, I was wondering if it would be odd for those who
> > >
> > > can
> > >
> > > only
> > >
> > > see
> > >
> > > the
> > >
> > > producer side to set configs for replication
> > >
> > > factor/partitions
> > >
> > > and
> > >
> > > see
> > >
> > > different results. Currently the documentation for the
> > >
> > > config
> > >
> > > states
> > >
> > > that
> > >
> > > the config values are only used when the broker config is
> > >
> > > not
> > >
> > > enabled,
> > >
> > > but
> > >
> > > this might not always be clear to the user. Changing the
> > >
> > > code
> > >
> > > to
> > >
> > > have
> > >
> > > the
> > >
> > > producer's configurations take precedence is possible, but
> > >
> > > I
> > >
> > > was
> > >
> > > wondering
> > >
> > > what everyone thought.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Just a quick update--
> > >
> > > It seems that enabling both the broker and producer
> > >
> > > configs
> > >
> > > works
> > >
> > > fine,
> > >
> > > except that the broker configurations for partitions,
> > >
> > > replication
> > >
> > > factor
> > >
> > > take precedence.
> > > I don't know if that is something we would want to
> > >
> > > change, but
> > >
> > > I'll be
> > >
> > > updating the KIP for now to reflect this. Perhaps we would
> > >
> > > want to
> > >
> > > add more
> > >
> > > to the documentation of the the producer configs to
> > >
> > > clarify.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for looking at the KIP. I can definitely add to
> > >
> > > the
> > >
> > > title
> > >
> > > to
> > >
> > > make
> > >
> > > it more clear.
> > >
> > > It makes sense that both configurations could be turned
> > >
> > > on
> > >
> > > since
> > >
> > > there
> > >
> > > are many cases where the user can not control the
> > >
> > > server-side
> > >
> > > configurations. I was a little concerned about how both
> > >
> > > interacting
> > >
> > > would
> > >
> > > work out -- if there would be to many requests for new
> > >
> > > topics,
> > >
> > > for
> > >
> > > example.
> > >
> > > But it since it does make sense to allow both
> > >
> > > configurations
> > >
> > > enabled, I
> > >
> > > will test out some scenarios and I'll change the KIP to
> > >
> > > support
> > >
> > > this.
> > >
> > > I also agree with documentation about distinguishing the
> > >
> > > differences. I
> > >
> > > was having some trouble with the wording but I like the
> > >
> > > phrases
> > >
> > > "server-side" and "client-side." That's a good
> > >
> > > distinction I
> > >
> > > can
> > >
> > > use
> > >
> > > when
> > >
> > > describing.
> > >
> > > I'll try to update the KIP soon keeping everyone's input
> > >
> > > in
> > >
> > > mind.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> > >
> > > cmccabe@ apache. org ( cmccabe@apache.org )
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP. This seems like a good step towards
> > >
> > > removing
> > >
> > > server-side topic auto-creation.
> > >
> > > We should add included "client-side" to the title of
> > >
> > > the KIP
> > >
> > > somewhere,
> > >
> > > to make it clear that we're talking about client-side
> > >
> > > auto
> > >
> > > creation.
> > >
> > > The KIP says:
> > >
> > > In order to automatically create topics with the
> > >
> > > producer, the
> > >
> > > producer's
> > >
> > > auto.create.topics.enable config must be set to true
> > >
> > > and
> > >
> > > the
> > >
> > > broker
> > >
> > > config should be set to false
> > >
> > > From a user's point of view, this seems
> > >
> > > counter-intuitive.
> > >
> > > In
> > >
> > > order to
> > >
> > > auto-create topics the broker's
> > >
> > > auto.create.topics.enable
> > >
> > > config
> > >
> > > should be
> > >
> > > set to false? It seems like the server-side
> > >
> > > auto-create is
> > >
> > > unrelated to
> > >
> > > the client-side auto-create. We could have both turned
> > >
> > > on
> > >
> > > (and
> > >
> > > I'm
> > >
> > > sure
> > >
> > > that in the real world, people will try this
> > >
> > > configuration...)
> > >
> > > There's no
> > >
> > > reason not to support this, I think.
> > >
> > > We should add some documentation explaining the
> > >
> > > difference
> > >
> > > between
> > >
> > > server-side and client-side auto-creation. Without
> > >
> > > documentation,
> > >
> > > an admin
> > >
> > > might think that they had disabled all forms of
> > >
> > > auto-creation by
> > >
> > > setting
> > >
> > > the -side setting to false-- but this is not the case,
> > >
> > > of
> > >
> > > course.
> > >
> > > best,
> > > Colin
> > >
> > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > >
> > > Hi Dhruvil,
> > >
> > > Thanks for reading the KIP!
> > > That was the general idea for deprecation. We would
> > >
> > > log a
> > >
> > > warning
> > >
> > > when the
> > >
> > > config is enabled on the broker.
> > > I also believe that there would be a change to
> > >
> > > documentation.
> > >
> > > If there is anything else that should be done, please
> > >
> > > let
> > >
> > > me
> > >
> > > know!
> > >
> > > Justine
> > >
> > > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> > >
> > > dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP, this is great!
> > >
> > > Could you add some more information about what
> > >
> > > deprecating
> > >
> > > the
> > >
> > > broker
> > >
> > > configuration means? Would we log a warning in the
> > >
> > > logs
> > >
> > > when
> > >
> > > auto
> > >
> > > topic
> > >
> > > creation is enabled on the broker, for example?
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> > >
> > > jolshan@ confluent. io ( jolshan@confluent.io ) >
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I'd like to start a discussion thread for KIP-487. This KIP plans
> > >
> > > to
> > >
> > > deprecate the current system of
> > >
> > > auto-creating
> > >
> > > topics
> > >
> > > through requests to the metadata and give the
> > >
> > > producer the
> > >
> > > ability to
> > >
> > > automatically create topics instead.
> > >
> > > More information can be found here:
> > >
> > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> > https://cwiki.
> > > apache.org/confluence/display/KAFKA/ )
> > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > >
> > > Thank you,
> > > Justine Olshan
> > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Justine Olshan <jo...@confluent.io>.

Hi Harsha,

I think my message may have gotten lost in all the others.

Two of the goals of this KIP are to 1) allow auto-creation on specific
clients when the broker default is false and 2) eventually replace the
broker config.

In order to accomplish these two goals, we need the producer to be able to
create topics despite the broker config. (How can we replace a function
when we rely on it?)
I think at this point we have a fundamental disagreement in what we should
allow the producer to do.
In my previous message I mentioned a config that would allow for the broker
to prevent producer auto-creation. (It would be disabled by default.) It
would fix your issue for now, but could lead to more complications later.

Thank you,
Justine


On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io> wrote:

> On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz> wrote:
>
> > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> >
> > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmccabe@apache.org >
> > wrote:
> >
> > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> >
> > Hi Colin,
> > "Hmm... I'm not sure I follow. Users don't have to build their own
> > tooling, right? They can use any of the shell scripts that we've shipped
> in
> > the last few releases. For example, if any of your users run it, this
> shell
> > script will delete all of the topics from your non-security-enabled
> > cluster:
> >
> > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > --bootstrap-server localhost:9092 --list 2>/dev/null
> > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > --bootstrap-server localhost:9092 --delete
> > --topic
> >
> > They will need to fill in the correct bootstrap servers list, of course,
> > not localhost. This deletion script will work on some pretty old brokers,
> > even back to the 0.10 releases. It seems a little odd to trust your users
> > with this power, but not trust them to avoid changing a particular
> > configuration key."
> >
> > The above will blocked by the server if we set delete.topic.enable to
> > false and thats exactly what I am asking for.
> >
> > Hi Harsha,
> >
> > I was wondering if someone was going to bring up that configuration :)
> >
> > it's an interesting complication, but globally disabling topic deletion
> is
> > not very practical for most use-cases.
> >
> > In any case, there are plenty of other bad things that users with full
> > permissions can do that aren't blocked by any server configuration. For
> > example, they can delete every record in every topic. I can write a
> script
> > for that too, and there's no server configuration you can set to disable
> > it. Or I could simply create hundreds of thousands of topics, until
> cluster
> > performance becomes unacceptable (this will be even more of a problem if
> > someone configured delete.topic.enable as false). Or publish bad data to
> > every topic, etc. etc.
> >
> > The point I'm trying to make here is that you can't rely on these kind of
> > server-side configurations for security. At most, they're a way to set up
> > certain very simple policies. But the policies are so simple that they're
> > hardly ever useful any more.
> >
> > For example, if the problem you want to solve is that you want a user to
> > only be able to create 50 topics and not delete anyone else's topics, you
> > can solve that with a CreateTopicsPolicy that limits the number of
> topics,
> > and some ACLs. There's no combination of auto.create.topics.enable and
> > delete.topic.enable that will help here.
> >
> > Hi Colin,
> >
> >             Well you gave the example that a user can delete the topics
> > just by running that script  :).
> >
> > I understand there are open APIs in Kafka and can lead to rogue clients
> > taking advantage of it without proper security in place.
> >
> > What I am asking so far in this thread is , this KIP is changing the
> > producer behavior and its not backward compatible.
> >
> > The change is backwards compatible. The default will still be server-side
> > topic auto-creation, just like now.
> >
> You will have to specifically change the producer config to get the new
> > behavior.
> >
>
>
> I disagree.  Today server can turn off the topic creation  neither producer
> or consumer can create a topic. With this KIP , producer can create a topic
> by turning on client side config when server side config is turned off.
>
>
> We can still achieve
> > the main goal of the KIP which is to change MetadataRequest  creating
> > topics and send a CreateTopicRequest from Producer and also keep the
> server
> > side config to have precedence.  This KIP originally written to have
> server
> > side preference and there is not much explanation why it changed to have
> > Producer side preference.
> >
> > Arguing that AdminClient can do that and so we are going to make Producer
> > do the same doesn't make sense.
> >
> > "The downside is that if we wanted to check a server side configuration
> > before sending the create topics request, the code would be more complex.
> > The behavior would also not be consistent with how topic auto-creation is
> > handled in Kafka Streams."
> >
> > I am not sure why you need to check server side configuration before
> > sending create topics request. A user enables producer side config to
> > create topics.
> > Producer sends a request to the broker and if the broker has
> > auto.topic.create.enable to true (default) it will allow creation of
> > topics. If it set to false it returns error back to the client.
> >
> > auto.topic.create.enable has never affected CreateTopicsRequest. If you
> > submit a CreateTopicsRequest and you are authorized, a topic will be
> > created, regardless of what the value of auto.topic.create.enable is.
> This
> > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?)
> >
> > I don't see how this behavior will be different in Kafka streams. By
> > default server allows the topic creation and with this KIP, It will only
> > allow creation of topic when both producer and server side are turned on.
> > Its exactly the same behavior in KIP-361.
> >
> > I suggest running a streams job as a test. It creates the topics it needs
> > using AdminClient. The value of auto.topic.create.enable is not relevant.
> > Whether it is true or false, the topics will still be created.
> >
> > "In general, it would be nice if we could keep the security and access
> > control model simple and not introduce a lot of special cases and
> > exceptions. Kafka has basically converged on using ACLs and
> > CreateTopicPolicy classes to control who can create topics and where.
> > Adding more knobs seems like a step backwards, especially when the
> proposed
> > knobs don't work consistently across components, and don't provide true
> > security." This is not about access control at all. Shipping sane
> defaults
> > should be prioritized.
> >
> > I don't think the default is really the question here. I think everyone
> > agrees that the default for client-side auto-topic creation should be
> off.
> >
> > My point goes back to KIP-361 why we kept the server side config to have
> > preference. We can keep bringing back the discussion of security policies
> > but a producer client overiding server side setting is not just security
> > policy issue .
> >
> > "Producer client can override server side setting and not consumer and
> > also producer can only override creation of topic but no the replication
> > and partitions. " This doesn't make sense to me and why we want to
> > introduce such a confusing behavior.
> >
> > The scenarios that you're describing (such as dealing with a poorly
> > configured client that tries to create topics it should not) do seem to
> be
> > about access control.
> >
> > We keep talking about CreateTopicPolicy and yet we don't have default one
> > and asking every user of Kafka implement their own doesn't make sense
> here.
> >
> > The point of CreateTopicPolicy is exactly that you can implement your
> own,
> > though. It's a way of customizing what the policy is in your specific
> > cluster.
> >
> > I agree that most people don't want to write a custom policy. But most of
> > those people are satisfied with just setting up ACLs to set up simple
> > policies like this user can create topics, that user can't, etc. That's
> why
> > I suggested adding support for ACLs to insecure clusters might be
> helpful.
> >
> > Also, just as a side note, wouldn't it would be impossible for us to
> > specify a default CreateTopicsPolicy that did anything at this point
> anyway
> > without breaking backwards compatibility? Maybe I'm misunderstanding the
> > proposal.
> >
> > I am asking about exact behavior that we shipped for consumer side
> https:/
> > / cwiki. apache. org/ confluence/ display/ KAFKA/
> > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > (
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> > )
> >
> > I still didn't get any response why this behavior shouldn't be exactly
> > like Kafka consumer and why do we want to have producer to overrider
> server
> > side config and while not allowing consumer to do so. We are not even
> > allowing the same contract and this will cause more confusion from the
> > users standpoint.
> >
> > Hmm. The consumer already has a way to override the server side
> > configuration. See KIP-361: Add Consumer Configuration to Disable Auto
> > Topic Creation. This lets the consumer skip auto-creating topics, even if
> > auto-creation is enabled on the broker.
> >
> > Yes that's what I am saying and it doesn't allow topic creation if the
> > server side auto-creation is disabled and in consumer its enabled.   I
> > would like to see the exact behavior for producer as stated in KIP-361.
> >
> > To be fair, the KIP should probably discuss why we don't want to
> implement
> > client-side topic creation in the consumer in "rejected alternatives."
> > Maybe Justine can add more context here in the KIP.
> >
> > The last time we talked about this, the reasoning was that we wanted to
> > eventually get rid of consumer-side auto-topic creation entirely, and so
> it
> > wasn't worth implementing an improved way of doing it. Producer side auto
> > topic-creation, in contrast, will probably stick around, although we'd
> like
> > to deprecate and remove the problematic server-side mechanism over time.
> >
> > We should implement the producer side topic creation with proper create
> > topic request  and I am in favor of this KIP.
> >
> > All I am asking is don't make the producer to override server side config
> > and keep it similar to KIP-361 just like consumer, it honors what set on
> > server side which takes the precedence.   Changing this behavior in
> > producer is backward incompatible and will catch users by surprise.
> >
> > All arguments for enforcing producer side overriding config can still be
> > achieved. Server side auto creation turned on by default and  any new
> > producer client setting their auto creation config to true will create
> > topics in default behavior and any user who set server side to false and
> > upgrades to latest kafka with this KIP will not see any unwanted behavior
> > from clients.  IMO this is not a unreasonable request on this KIP and
> this
> > requested behavior is exactly what the KIP initially proposed.
> >
> > Thanks,
> >
> > Harsha
> >
> > best,
> > Colin
> >
> > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache. org (
> > cmccabe@apache.org ) > wrote:
> >
> > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> >
> > Not sure how the AdminClient applies here, It is an external API and is
> > not part of KafkaProducer so any user who updates to latest version of
> > Kafka with this feature get to create the topics.
> > They have to build a tooling around AdminClient allow themselves to
> >
> > create
> >
> > topics.
> >
> > Hi Harsha,
> >
> > Hmm... I'm not sure I follow. Users don't have to build their own
> tooling,
> > right? They can use any of the shell scripts that we've shipped in the
> last
> > few releases. For example, if any of your users run it, this shell script
> > will delete all of the topics from your non-security-enabled cluster:
> >
> > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > --bootstrap-server localhost:9092 --list 2>/dev/null
> > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > --bootstrap-server localhost:9092 --delete
> > --topic
> >
> > They will need to fill in the correct bootstrap servers list, of course,
> > not localhost. This deletion script will work on some pretty old brokers,
> > even back to the 0.10 releases. It seems a little odd to trust your users
> > with this power, but not trust them to avoid changing a particular
> > configuration key.
> >
> > There is no behavior in Kafka producer that allowed users to delete the
> > topics or delete the records. So citing them as an example doesn't makes
> > sense in this context.
> >
> > I think Kafka Streams is relevant here. After all, it's software that we
> > ship as part of the official Kafka release. And it auto-creates topics
> even
> > when auto.create.topics.enable is set to false on the broker.
> >
> > I think that auto.create.topics.enable was never intended as a security
> > setting to control access. It was intended as a way to disable the
> > broker-side auto-create feature specifically, because there were some
> > problems with that specific feature. Broker-side auto-creation is
> > frustrating because it's on by default, and because it can auto-create
> > topics even if the producer or consumer didn't explicitly ask for that.
> > Neither of those problems applies to this KIP: producers have to
> > specifically opt in, and it won't be on by default. Basically, we think
> > that client-side auto-creation is actually a lot better-- hence this KIP
> > :)
> >
> > But there is
> > a functionality which allowed creation of topics if they don't exist in
> >
> > the
> >
> > cluster and this behavior could be controlled by a config on the server
> > side. Now with this KIP we are allowing producer to make that decision
> > without any gateway on the server via configs. Any user who is not aware
> >
> > of
> >
> > such changes
> > can accidentally create these topics and we are essentially removing a
> > config that exists in brokers today to block this accidental creation and
> > allowing clients to take control.
> >
> > Again, I hope I'm not misinterpreting, but I don't see how this can be
> > turned on accidentally. The user would have to specifically turn this on
> in
> > the producer by setting the configuration key.
> >
> > I still didn't get any positives of not having server side configs? if
> you
> > want to turn them on and allow any client to create topics set the
> default
> > to true
> > and allow users who doesn't want to have this feature let them turn them
> > off. It will be the exact behavior as it is today , as far as producer is
> > concerned. I am not
> > understanding why not having server side configs to gateways such a hard
> > requirement and this behavior exists today. As far I am concerned this
> > breaks the backward compatibility.
> >
> > The downside is that if we wanted to check a server side configuration
> > before sending the create topics request, the code would be more complex.
> > The behavior would also not be consistent with how topic auto-creation is
> > handled in Kafka Streams.
> >
> > In general, it would be nice if we could keep the security and access
> > control model simple and not introduce a lot of special cases and
> > exceptions. Kafka has basically converged on using ACLs and
> > CreateTopicPolicy classes to control who can create topics and where.
> > Adding more knobs seems like a step backwards, especially when the
> proposed
> > knobs don't work consistently across components, and don't provide true
> > security.
> >
> > Maybe we should support an insecure mode where there are still users and
> > ACLs. That would help people who don't want to set up Kerberos, etc. but
> > still want to protect against misconfigured clients or accidents. Hadoop
> > has something like this, and I think it was useful. NFS also supported
> > (supports?) a mode where you just pass whatever user ID you want and the
> > system believes you. These things clearly don't protect against malicious
> > users, but they can help set up policies when needed.
> >
> > best,
> > Colin
> >
> > Thanks,
> > Harsha
> >
> > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache. org (
> > cmccabe@apache.org ) > wrote:
> >
> > Hi Harsha,
> >
> > Thanks for taking a look.
> >
> > I'm not sure I follow the discussion about AdminClient.
> >
> > KafkaAdminClient
> >
> > has been around for about 2 years at this point as a public class.
> >
> > There
> >
> > are many programs that use it to automatically create topics. Kafka
> > Streams does this, for example. If any of your users run Kafka
> >
> > Streams,
> >
> > they will be auto-creating topics, regardless of what setting you use
> >
> > for
> >
> > auto.create.topics.enable.
> >
> > A big part of the annoyance of auto-topic creation right now is that
> >
> > it is
> >
> > on by default. The new configuration proposed by KIP-487 wouldn't be.
> > Users would have to explicitly opt in to the new behavior of
> >
> > client-side
> >
> > topic creation. If you run without security, you're already putting a
> >
> > huge
> >
> > amount of trust in your users. For example, you trust them not to
> >
> > delete
> >
> > records with the kafka-delete-records. sh ( http://kafka-delete-records.
> > sh/
> > ) command, or delete topics with kafka-topics. sh ( http://kafka-topics.
> > sh/
> > ). Trusting them not to set a certain config value seems minor in
> > comparison, right?
> >
> > best,
> > Colin
> >
> > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> >
> > Hi,
> > Even with policies one needs to implement that, so for every
> >
> > user who
> >
> > doesn't want a producer to create topics or have limits around
> >
> > partitions &
> >
> > replication factor they have to implement a policy. The KIP is changing
> > the behavior , it might not be introducing
> >
> > the
> >
> > new functionality but it will enable producers to override the create
> >
> > topic
> >
> > config settings on the broker. What I am asking for to provide a
> >
> > config
> >
> > that will disable auto creation of topics and if its enabled set some
> >
> > sane
> >
> > defaults so that clients can create a topic with in those limits. I
> >
> > don't
> >
> > see how this not related to this KIP.
> > If the server config options as I suggested doesn't interest
> >
> > you at
> >
> > least have a default CreateTopicPolices in place.
> > To give an example, In our environment we disable the
> > auto.create.topic.enable and force users to go through a centralized
> > service as we want capture more details about the topic creation and
> > requirements. With this KIP, a producer can create a topic with no
> >
> > bounds.
> >
> > Another example max.message.size we define that at cluster level
> >
> > and one
> >
> > can override max.messsage.size at topic level, any misconfiguration
> >
> > at
> >
> > this
> >
> > will cause service degradation. Its not always about the rogue
> >
> > clients,
> >
> > users can easily misconfigure and can cause an outage. Again we can talk
> > about CreateTopicPolicy but without having a
> >
> > default
> >
> > implementation and asking everyone to implement their own while
> >
> > changing
> >
> > the behavior in producer doesn't make sense to me.
> >
> > Thanks,
> > Harsha
> >
> > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me. uk (
> > ismael@juma.me.uk ) >
> >
> > wrote:
> >
> > Hi Harsha,
> >
> > I mentioned policies and the authorizer. For example, with
> > CreateTopicPolicy, you can implement the limits you describe. If
> >
> > you
> >
> > have
> >
> > ideas of how that should be improved, please submit a KIP. My
> >
> > point is
> >
> > that
> >
> > this KIP is not introducing any new functionality with regards to
> >
> > what
> >
> > rogue clients can do. It's using the existing protocol that is
> >
> > already
> >
> > exposed via the AdminClient. So, I don't think we need to address
> >
> > it in
> >
> > this KIP. Does that make sense?
> >
> > Ismael
> >
> > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> >
> > kafka@ harsha. io ( kafka@harsha.io ) >
> >
> > wrote:
> >
> > Ismael,
> > Sure AdminClient can do that and we should've shipped a config or
> >
> > use
> >
> > the
> >
> > existing one to block that. Not all users are yet to upgrade to
> >
> > AdminClient
> >
> > and start using that to cause issues yet. In shared environment we
> >
> > should
> >
> > allow server to set sane defaults and not allow every client to go
> >
> > ahead
> >
> > create random no.of topic/partitions and replication factor. Even
> >
> > if
> >
> > the
> >
> > users want to allow topic creation proposed in the KIP , it makes
> >
> > sense to
> >
> > have some guards against the no.of partitions and replication
> >
> > factor.
> >
> > Authorizer is not always an answer to block requests and having
> >
> > users
> >
> > set
> >
> > server side configs to protect a multi-tenant environment is
> >
> > required.
> >
> > In a
> >
> > non-secure environment Authorizer is a blunt instrument either you
> >
> > end
> >
> > up
> >
> > blocking everyone or allowing everyone.
> > I am asking to have server side that allow clients to create
> >
> > topics or
> >
> > not
> >
> > , if they are allowed set a ceiling on max no.of partitions and
> > replication-factor.
> >
> > -Harsha
> >
> > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com ( ismaelj@gmail.com
> > )
> >
> > wrote:
> >
> > Harsha,
> >
> > Rogue clients can use the admin client to create topics and
> >
> > partitions.
> >
> > ACLs and policies can help in that case as well as this one.
> >
> > Ismael
> >
> > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha. io (
> > kafka@harsha.io ) >
> >
> > wrote:
> >
> > Hi Justine,
> > Thanks for the KIP.
> > "When server-side auto-creation is disabled, client-side
> >
> > auto-creation
> >
> > will try to use client-side configurations"
> > If I understand correctly, this KIP is removing any server-side
> >
> > blocking
> >
> > client auto creation of topic?
> > if so this will present potential issue of rogue client creating
> >
> > ton of
> >
> > topic-partitions and potentially bringing down the service for
> >
> > everyone
> >
> > or
> >
> > degrade the service itself.
> > By reading the KIP its not clear to me that there is a clear way to
> >
> > block
> >
> > auto creation topics of all together from clients by server side
> >
> > config.
> >
> > Server side configs of default topic, partitions should take higher
> > precedence and client shouldn't be able to create a topic with
> >
> > higher
> >
> > no.of
> >
> > partitions, replication than what server config specifies.
> >
> > Thanks,
> > Harsha
> >
> > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Hi all,
> > I made some changes to the KIP. Hopefully this configuration change
> >
> > will
> >
> > make things a little clearer.
> >
> > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> https://cwiki.
> > apache.org/confluence/display/KAFKA/ )
> > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> >
> > Please let me know if you have any feedback or questions!
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache. org (
> > cmccabe@apache.org ) >
> >
> > wrote:
> >
> > Hi Mickael,
> >
> > I think you bring up a good point. It would be better if we didn't
> >
> > ever
> >
> > have to set up client-side configuration for this feature, and
> >
> > KIP-464
> >
> > would let us skip this entirely.
> >
> > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> >
> > Hi Mickael,
> > I agree that KIP-464 works on newer brokers, but I was a bit
> >
> > worried
> >
> > how
> >
> > things would play out on older brokers that* do not *have KIP 464
> >
> > included.
> >
> > Is it enough to throw an error in this case when producer configs
> >
> > are
> >
> > not
> >
> > specified?
> >
> > I think the right thing to do would be to log an error message in
> >
> > the
> >
> > client. We will need to have that capability in any case, to cover
> > scenarios like the client trying to auto-create a topic that they
> >
> > don't
> >
> > have permission to create. Or a client trying to create a topic on
> >
> > a
> >
> > broker
> >
> > so old that CreateTopicsRequest is not supported.
> >
> > The big downside to relying on KIP-464 is that it is a very recent
> >
> > feature
> >
> > -- so recent that it hasn't even made its way to any official
> >
> > Apache
> >
> > release. It's scheduled for the upcoming 2.4 release in a few
> >
> > months.
> >
> > So if you view this KIP as a step towards removing broker-side
> > auto-create, you might want to support older brokers just to
> >
> > accelerate
> >
> > adoption, and hasten the day when we can finally flip broker-side
> > auto-create to off (or even remove it entirely).
> >
> > I have to agree, though, that having client-side configurations for
> >
> > number
> >
> > of partitions and replication factor is messy. Maybe it would be
> >
> > worth
> >
> > it
> >
> > to restrict support to post-KIP-464 brokers, if we could avoid
> >
> > creating
> >
> > more configs.
> >
> > best,
> > Colin
> >
> > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> >
> > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> >
> > wrote:
> >
> > Hi Justine,
> >
> > We can rely on KIP-464 which allows to omit the partition count or
> > replication factor when creating a topic. In that case, the broker
> >
> > defaults
> >
> > are used.
> >
> > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Michael,
> > That makes sense to me!
> > To clarify, in the current state of the KIP, the producer does not
> >
> > rely
> >
> > on
> >
> > the broker to autocreate--if the broker's config is disabled, then
> >
> > the
> >
> > producer can autocreate on its own with a create topic request (the
> >
> > same
> >
> > type of request the admin client uses).
> > However, if both configs are enabled, the broker will autocreate
> >
> > through
> >
> > a
> >
> > metadata request before the producer gets a chance. Of course, the
> >
> > way
> >
> > to
> >
> > avoid this, is to do as you suggested, and set
> >
> > the
> >
> > "allow_auto_topic_creation" field to false.
> >
> > I think the only thing we need to be careful with in this setup is
> >
> > without
> >
> > KIP 464, we can not use broker defaults for this topic. A user
> >
> > needs
> >
> > to
> >
> > specify the number of partition and replication factor in the
> >
> > config.
> >
> > An
> >
> > alternative to this is to have coded defaults for when these
> >
> > configs
> >
> > are
> >
> > unspecified, but it is not immediately apparent what these defaults
> >
> > should
> >
> > be.
> >
> > Thanks again for reading my KIP,
> > Justine
> >
> > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> >
> > mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the response!
> > In my opinion, it would be better if the producer did not rely at
> >
> > all
> >
> > on the broker auto create feature as this is what we're aiming to
> > deprecate. When requesting metadata we can set the
> > "allow_auto_topic_creation" field to false to avoid the broker auto
> > creation. Then if the topic is not existing, send a
> >
> > CreateTopicRequest.
> >
> > What do you think?
> >
> > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Currently the way it is implemented, the broker auto-creation
> >
> > configuration
> >
> > takes precedence. The producer will not use the CreateTopics
> >
> > request.
> >
> > (Technically it can--but the topic will already be created
> >
> > through
> >
> > the
> >
> > broker, so it will never try to create the topic.) It is possible
> >
> > to
> >
> > change this however, and I'd be happy to
> >
> > discuss
> >
> > the
> >
> > benefits of this alternative.
> >
> > Thank you,
> > Justine
> >
> > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> >
> > mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP!
> >
> > In case auto creation is enabled on both the client and server,
> >
> > will
> >
> > the producer still use the AdminClient (CreateTopics request)
> >
> > to
> >
> > create topics? and not rely on the broker auto create. I'm
> >
> > guessing the
> >
> > answer is yes but can you make it explicit.
> >
> > Thank you
> >
> > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Hi,
> > Just a friendly reminder to take a look at this KIP if you
> >
> > have
> >
> > the
> >
> > time.
> >
> > I was thinking about broker vs. client default precedence,
> >
> > and I
> >
> > think it
> >
> > makes sense to keep the broker as the default used when both
> >
> > client-side
> >
> > and broker-side defaults are configured. The idea is that
> >
> > there
> >
> > would be
> >
> > pretty clear documentation, and that many systems with
> >
> > configurations
> >
> > that
> >
> > the client could not change would likely have the auto-create
> >
> > default
> >
> > off.
> >
> > (In cloud for example).
> >
> > It also seems like in most cases, the consumer config
> > ' allow. auto. create. topics ( http://allow.auto.create.topics/ ) ' was
> > created to actually prevent
> >
> > the
> >
> > creation
> >
> > of
> >
> > topics, so the loss of creation functionality will not be a
> >
> > big
> >
> > problem.
> >
> > I'm happy to discuss any other compatibility problems or
> >
> > components
> >
> > of
> >
> > this KIP.
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io )
> >
> > wrote:
> >
> > Hello all,
> >
> > I was looking at this KIP again, and there is a decision I
> >
> > made
> >
> > that I
> >
> > think is worth discussing.
> >
> > In the case where both the broker and producer's
> > 'auto.create.topics.enable' are set to true, we have to
> >
> > choose
> >
> > either
> >
> > the
> >
> > broker configs or the producer configs for the replication
> > factor/partitions.
> >
> > Currently, the decision is to have the broker defaults take
> >
> > precedence.
> >
> > (It is easier to do this in the implementation.) It also
> >
> > makes
> >
> > some
> >
> > sense
> >
> > for this behavior to take precedence since this behavior
> >
> > already
> >
> > occurs as
> >
> > the default.
> >
> > However, I was wondering if it would be odd for those who
> >
> > can
> >
> > only
> >
> > see
> >
> > the
> >
> > producer side to set configs for replication
> >
> > factor/partitions
> >
> > and
> >
> > see
> >
> > different results. Currently the documentation for the
> >
> > config
> >
> > states
> >
> > that
> >
> > the config values are only used when the broker config is
> >
> > not
> >
> > enabled,
> >
> > but
> >
> > this might not always be clear to the user. Changing the
> >
> > code
> >
> > to
> >
> > have
> >
> > the
> >
> > producer's configurations take precedence is possible, but
> >
> > I
> >
> > was
> >
> > wondering
> >
> > what everyone thought.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Just a quick update--
> >
> > It seems that enabling both the broker and producer
> >
> > configs
> >
> > works
> >
> > fine,
> >
> > except that the broker configurations for partitions,
> >
> > replication
> >
> > factor
> >
> > take precedence.
> > I don't know if that is something we would want to
> >
> > change, but
> >
> > I'll be
> >
> > updating the KIP for now to reflect this. Perhaps we would
> >
> > want to
> >
> > add more
> >
> > to the documentation of the the producer configs to
> >
> > clarify.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Hi Colin,
> >
> > Thanks for looking at the KIP. I can definitely add to
> >
> > the
> >
> > title
> >
> > to
> >
> > make
> >
> > it more clear.
> >
> > It makes sense that both configurations could be turned
> >
> > on
> >
> > since
> >
> > there
> >
> > are many cases where the user can not control the
> >
> > server-side
> >
> > configurations. I was a little concerned about how both
> >
> > interacting
> >
> > would
> >
> > work out -- if there would be to many requests for new
> >
> > topics,
> >
> > for
> >
> > example.
> >
> > But it since it does make sense to allow both
> >
> > configurations
> >
> > enabled, I
> >
> > will test out some scenarios and I'll change the KIP to
> >
> > support
> >
> > this.
> >
> > I also agree with documentation about distinguishing the
> >
> > differences. I
> >
> > was having some trouble with the wording but I like the
> >
> > phrases
> >
> > "server-side" and "client-side." That's a good
> >
> > distinction I
> >
> > can
> >
> > use
> >
> > when
> >
> > describing.
> >
> > I'll try to update the KIP soon keeping everyone's input
> >
> > in
> >
> > mind.
> >
> > Thanks,
> > Justine
> >
> > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> >
> > cmccabe@ apache. org ( cmccabe@apache.org )
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP. This seems like a good step towards
> >
> > removing
> >
> > server-side topic auto-creation.
> >
> > We should add included "client-side" to the title of
> >
> > the KIP
> >
> > somewhere,
> >
> > to make it clear that we're talking about client-side
> >
> > auto
> >
> > creation.
> >
> > The KIP says:
> >
> > In order to automatically create topics with the
> >
> > producer, the
> >
> > producer's
> >
> > auto.create.topics.enable config must be set to true
> >
> > and
> >
> > the
> >
> > broker
> >
> > config should be set to false
> >
> > From a user's point of view, this seems
> >
> > counter-intuitive.
> >
> > In
> >
> > order to
> >
> > auto-create topics the broker's
> >
> > auto.create.topics.enable
> >
> > config
> >
> > should be
> >
> > set to false? It seems like the server-side
> >
> > auto-create is
> >
> > unrelated to
> >
> > the client-side auto-create. We could have both turned
> >
> > on
> >
> > (and
> >
> > I'm
> >
> > sure
> >
> > that in the real world, people will try this
> >
> > configuration...)
> >
> > There's no
> >
> > reason not to support this, I think.
> >
> > We should add some documentation explaining the
> >
> > difference
> >
> > between
> >
> > server-side and client-side auto-creation. Without
> >
> > documentation,
> >
> > an admin
> >
> > might think that they had disabled all forms of
> >
> > auto-creation by
> >
> > setting
> >
> > the -side setting to false-- but this is not the case,
> >
> > of
> >
> > course.
> >
> > best,
> > Colin
> >
> > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> >
> > Hi Dhruvil,
> >
> > Thanks for reading the KIP!
> > That was the general idea for deprecation. We would
> >
> > log a
> >
> > warning
> >
> > when the
> >
> > config is enabled on the broker.
> > I also believe that there would be a change to
> >
> > documentation.
> >
> > If there is anything else that should be done, please
> >
> > let
> >
> > me
> >
> > know!
> >
> > Justine
> >
> > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> >
> > dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP, this is great!
> >
> > Could you add some more information about what
> >
> > deprecating
> >
> > the
> >
> > broker
> >
> > configuration means? Would we log a warning in the
> >
> > logs
> >
> > when
> >
> > auto
> >
> > topic
> >
> > creation is enabled on the broker, for example?
> >
> > Thanks,
> > Dhruvil
> >
> > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> >
> > jolshan@ confluent. io ( jolshan@confluent.io ) >
> >
> > wrote:
> >
> > Hello all,
> >
> > I'd like to start a discussion thread for KIP-487. This KIP plans
> >
> > to
> >
> > deprecate the current system of
> >
> > auto-creating
> >
> > topics
> >
> > through requests to the metadata and give the
> >
> > producer the
> >
> > ability to
> >
> > automatically create topics instead.
> >
> > More information can be found here:
> >
> > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> https://cwiki.
> > apache.org/confluence/display/KAFKA/ )
> > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> >
> > Thank you,
> > Justine Olshan
> >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Harsha Chintalapani <ka...@harsha.io>.

On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz> wrote:

> On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
>
> On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmccabe@apache.org >
> wrote:
>
> On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
>
> Hi Colin,
> "Hmm... I'm not sure I follow. Users don't have to build their own
> tooling, right? They can use any of the shell scripts that we've shipped in
> the last few releases. For example, if any of your users run it, this shell
> script will delete all of the topics from your non-security-enabled
> cluster:
>
> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> --bootstrap-server localhost:9092 --list 2>/dev/null
> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> --bootstrap-server localhost:9092 --delete
> --topic
>
> They will need to fill in the correct bootstrap servers list, of course,
> not localhost. This deletion script will work on some pretty old brokers,
> even back to the 0.10 releases. It seems a little odd to trust your users
> with this power, but not trust them to avoid changing a particular
> configuration key."
>
> The above will blocked by the server if we set delete.topic.enable to
> false and thats exactly what I am asking for.
>
> Hi Harsha,
>
> I was wondering if someone was going to bring up that configuration :)
>
> it's an interesting complication, but globally disabling topic deletion is
> not very practical for most use-cases.
>
> In any case, there are plenty of other bad things that users with full
> permissions can do that aren't blocked by any server configuration. For
> example, they can delete every record in every topic. I can write a script
> for that too, and there's no server configuration you can set to disable
> it. Or I could simply create hundreds of thousands of topics, until cluster
> performance becomes unacceptable (this will be even more of a problem if
> someone configured delete.topic.enable as false). Or publish bad data to
> every topic, etc. etc.
>
> The point I'm trying to make here is that you can't rely on these kind of
> server-side configurations for security. At most, they're a way to set up
> certain very simple policies. But the policies are so simple that they're
> hardly ever useful any more.
>
> For example, if the problem you want to solve is that you want a user to
> only be able to create 50 topics and not delete anyone else's topics, you
> can solve that with a CreateTopicsPolicy that limits the number of topics,
> and some ACLs. There's no combination of auto.create.topics.enable and
> delete.topic.enable that will help here.
>
> Hi Colin,
>
>             Well you gave the example that a user can delete the topics
> just by running that script  :).
>
> I understand there are open APIs in Kafka and can lead to rogue clients
> taking advantage of it without proper security in place.
>
> What I am asking so far in this thread is , this KIP is changing the
> producer behavior and its not backward compatible.
>
> The change is backwards compatible. The default will still be server-side
> topic auto-creation, just like now.
>
You will have to specifically change the producer config to get the new
> behavior.
>


I disagree.  Today server can turn off the topic creation  neither producer
or consumer can create a topic. With this KIP , producer can create a topic
by turning on client side config when server side config is turned off.


We can still achieve
> the main goal of the KIP which is to change MetadataRequest  creating
> topics and send a CreateTopicRequest from Producer and also keep the server
> side config to have precedence.  This KIP originally written to have server
> side preference and there is not much explanation why it changed to have
> Producer side preference.
>
> Arguing that AdminClient can do that and so we are going to make Producer
> do the same doesn't make sense.
>
> "The downside is that if we wanted to check a server side configuration
> before sending the create topics request, the code would be more complex.
> The behavior would also not be consistent with how topic auto-creation is
> handled in Kafka Streams."
>
> I am not sure why you need to check server side configuration before
> sending create topics request. A user enables producer side config to
> create topics.
> Producer sends a request to the broker and if the broker has
> auto.topic.create.enable to true (default) it will allow creation of
> topics. If it set to false it returns error back to the client.
>
> auto.topic.create.enable has never affected CreateTopicsRequest. If you
> submit a CreateTopicsRequest and you are authorized, a topic will be
> created, regardless of what the value of auto.topic.create.enable is. This
> behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?)
>
> I don't see how this behavior will be different in Kafka streams. By
> default server allows the topic creation and with this KIP, It will only
> allow creation of topic when both producer and server side are turned on.
> Its exactly the same behavior in KIP-361.
>
> I suggest running a streams job as a test. It creates the topics it needs
> using AdminClient. The value of auto.topic.create.enable is not relevant.
> Whether it is true or false, the topics will still be created.
>
> "In general, it would be nice if we could keep the security and access
> control model simple and not introduce a lot of special cases and
> exceptions. Kafka has basically converged on using ACLs and
> CreateTopicPolicy classes to control who can create topics and where.
> Adding more knobs seems like a step backwards, especially when the proposed
> knobs don't work consistently across components, and don't provide true
> security." This is not about access control at all. Shipping sane defaults
> should be prioritized.
>
> I don't think the default is really the question here. I think everyone
> agrees that the default for client-side auto-topic creation should be off.
>
> My point goes back to KIP-361 why we kept the server side config to have
> preference. We can keep bringing back the discussion of security policies
> but a producer client overiding server side setting is not just security
> policy issue .
>
> "Producer client can override server side setting and not consumer and
> also producer can only override creation of topic but no the replication
> and partitions. " This doesn't make sense to me and why we want to
> introduce such a confusing behavior.
>
> The scenarios that you're describing (such as dealing with a poorly
> configured client that tries to create topics it should not) do seem to be
> about access control.
>
> We keep talking about CreateTopicPolicy and yet we don't have default one
> and asking every user of Kafka implement their own doesn't make sense here.
>
> The point of CreateTopicPolicy is exactly that you can implement your own,
> though. It's a way of customizing what the policy is in your specific
> cluster.
>
> I agree that most people don't want to write a custom policy. But most of
> those people are satisfied with just setting up ACLs to set up simple
> policies like this user can create topics, that user can't, etc. That's why
> I suggested adding support for ACLs to insecure clusters might be helpful.
>
> Also, just as a side note, wouldn't it would be impossible for us to
> specify a default CreateTopicsPolicy that did anything at this point anyway
> without breaking backwards compatibility? Maybe I'm misunderstanding the
> proposal.
>
> I am asking about exact behavior that we shipped for consumer side https:/
> / cwiki. apache. org/ confluence/ display/ KAFKA/
> KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> (
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> )
>
> I still didn't get any response why this behavior shouldn't be exactly
> like Kafka consumer and why do we want to have producer to overrider server
> side config and while not allowing consumer to do so. We are not even
> allowing the same contract and this will cause more confusion from the
> users standpoint.
>
> Hmm. The consumer already has a way to override the server side
> configuration. See KIP-361: Add Consumer Configuration to Disable Auto
> Topic Creation. This lets the consumer skip auto-creating topics, even if
> auto-creation is enabled on the broker.
>
> Yes that's what I am saying and it doesn't allow topic creation if the
> server side auto-creation is disabled and in consumer its enabled.   I
> would like to see the exact behavior for producer as stated in KIP-361.
>
> To be fair, the KIP should probably discuss why we don't want to implement
> client-side topic creation in the consumer in "rejected alternatives."
> Maybe Justine can add more context here in the KIP.
>
> The last time we talked about this, the reasoning was that we wanted to
> eventually get rid of consumer-side auto-topic creation entirely, and so it
> wasn't worth implementing an improved way of doing it. Producer side auto
> topic-creation, in contrast, will probably stick around, although we'd like
> to deprecate and remove the problematic server-side mechanism over time.
>
> We should implement the producer side topic creation with proper create
> topic request  and I am in favor of this KIP.
>
> All I am asking is don't make the producer to override server side config
> and keep it similar to KIP-361 just like consumer, it honors what set on
> server side which takes the precedence.   Changing this behavior in
> producer is backward incompatible and will catch users by surprise.
>
> All arguments for enforcing producer side overriding config can still be
> achieved. Server side auto creation turned on by default and  any new
> producer client setting their auto creation config to true will create
> topics in default behavior and any user who set server side to false and
> upgrades to latest kafka with this KIP will not see any unwanted behavior
> from clients.  IMO this is not a unreasonable request on this KIP and this
> requested behavior is exactly what the KIP initially proposed.
>
> Thanks,
>
> Harsha
>
> best,
> Colin
>
> On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache. org (
> cmccabe@apache.org ) > wrote:
>
> On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
>
> Not sure how the AdminClient applies here, It is an external API and is
> not part of KafkaProducer so any user who updates to latest version of
> Kafka with this feature get to create the topics.
> They have to build a tooling around AdminClient allow themselves to
>
> create
>
> topics.
>
> Hi Harsha,
>
> Hmm... I'm not sure I follow. Users don't have to build their own tooling,
> right? They can use any of the shell scripts that we've shipped in the last
> few releases. For example, if any of your users run it, this shell script
> will delete all of the topics from your non-security-enabled cluster:
>
> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> --bootstrap-server localhost:9092 --list 2>/dev/null
> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> --bootstrap-server localhost:9092 --delete
> --topic
>
> They will need to fill in the correct bootstrap servers list, of course,
> not localhost. This deletion script will work on some pretty old brokers,
> even back to the 0.10 releases. It seems a little odd to trust your users
> with this power, but not trust them to avoid changing a particular
> configuration key.
>
> There is no behavior in Kafka producer that allowed users to delete the
> topics or delete the records. So citing them as an example doesn't makes
> sense in this context.
>
> I think Kafka Streams is relevant here. After all, it's software that we
> ship as part of the official Kafka release. And it auto-creates topics even
> when auto.create.topics.enable is set to false on the broker.
>
> I think that auto.create.topics.enable was never intended as a security
> setting to control access. It was intended as a way to disable the
> broker-side auto-create feature specifically, because there were some
> problems with that specific feature. Broker-side auto-creation is
> frustrating because it's on by default, and because it can auto-create
> topics even if the producer or consumer didn't explicitly ask for that.
> Neither of those problems applies to this KIP: producers have to
> specifically opt in, and it won't be on by default. Basically, we think
> that client-side auto-creation is actually a lot better-- hence this KIP
> :)
>
> But there is
> a functionality which allowed creation of topics if they don't exist in
>
> the
>
> cluster and this behavior could be controlled by a config on the server
> side. Now with this KIP we are allowing producer to make that decision
> without any gateway on the server via configs. Any user who is not aware
>
> of
>
> such changes
> can accidentally create these topics and we are essentially removing a
> config that exists in brokers today to block this accidental creation and
> allowing clients to take control.
>
> Again, I hope I'm not misinterpreting, but I don't see how this can be
> turned on accidentally. The user would have to specifically turn this on in
> the producer by setting the configuration key.
>
> I still didn't get any positives of not having server side configs? if you
> want to turn them on and allow any client to create topics set the default
> to true
> and allow users who doesn't want to have this feature let them turn them
> off. It will be the exact behavior as it is today , as far as producer is
> concerned. I am not
> understanding why not having server side configs to gateways such a hard
> requirement and this behavior exists today. As far I am concerned this
> breaks the backward compatibility.
>
> The downside is that if we wanted to check a server side configuration
> before sending the create topics request, the code would be more complex.
> The behavior would also not be consistent with how topic auto-creation is
> handled in Kafka Streams.
>
> In general, it would be nice if we could keep the security and access
> control model simple and not introduce a lot of special cases and
> exceptions. Kafka has basically converged on using ACLs and
> CreateTopicPolicy classes to control who can create topics and where.
> Adding more knobs seems like a step backwards, especially when the proposed
> knobs don't work consistently across components, and don't provide true
> security.
>
> Maybe we should support an insecure mode where there are still users and
> ACLs. That would help people who don't want to set up Kerberos, etc. but
> still want to protect against misconfigured clients or accidents. Hadoop
> has something like this, and I think it was useful. NFS also supported
> (supports?) a mode where you just pass whatever user ID you want and the
> system believes you. These things clearly don't protect against malicious
> users, but they can help set up policies when needed.
>
> best,
> Colin
>
> Thanks,
> Harsha
>
> On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache. org (
> cmccabe@apache.org ) > wrote:
>
> Hi Harsha,
>
> Thanks for taking a look.
>
> I'm not sure I follow the discussion about AdminClient.
>
> KafkaAdminClient
>
> has been around for about 2 years at this point as a public class.
>
> There
>
> are many programs that use it to automatically create topics. Kafka
> Streams does this, for example. If any of your users run Kafka
>
> Streams,
>
> they will be auto-creating topics, regardless of what setting you use
>
> for
>
> auto.create.topics.enable.
>
> A big part of the annoyance of auto-topic creation right now is that
>
> it is
>
> on by default. The new configuration proposed by KIP-487 wouldn't be.
> Users would have to explicitly opt in to the new behavior of
>
> client-side
>
> topic creation. If you run without security, you're already putting a
>
> huge
>
> amount of trust in your users. For example, you trust them not to
>
> delete
>
> records with the kafka-delete-records. sh ( http://kafka-delete-records.
> sh/
> ) command, or delete topics with kafka-topics. sh ( http://kafka-topics.
> sh/
> ). Trusting them not to set a certain config value seems minor in
> comparison, right?
>
> best,
> Colin
>
> On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
>
> Hi,
> Even with policies one needs to implement that, so for every
>
> user who
>
> doesn't want a producer to create topics or have limits around
>
> partitions &
>
> replication factor they have to implement a policy. The KIP is changing
> the behavior , it might not be introducing
>
> the
>
> new functionality but it will enable producers to override the create
>
> topic
>
> config settings on the broker. What I am asking for to provide a
>
> config
>
> that will disable auto creation of topics and if its enabled set some
>
> sane
>
> defaults so that clients can create a topic with in those limits. I
>
> don't
>
> see how this not related to this KIP.
> If the server config options as I suggested doesn't interest
>
> you at
>
> least have a default CreateTopicPolices in place.
> To give an example, In our environment we disable the
> auto.create.topic.enable and force users to go through a centralized
> service as we want capture more details about the topic creation and
> requirements. With this KIP, a producer can create a topic with no
>
> bounds.
>
> Another example max.message.size we define that at cluster level
>
> and one
>
> can override max.messsage.size at topic level, any misconfiguration
>
> at
>
> this
>
> will cause service degradation. Its not always about the rogue
>
> clients,
>
> users can easily misconfigure and can cause an outage. Again we can talk
> about CreateTopicPolicy but without having a
>
> default
>
> implementation and asking everyone to implement their own while
>
> changing
>
> the behavior in producer doesn't make sense to me.
>
> Thanks,
> Harsha
>
> On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me. uk (
> ismael@juma.me.uk ) >
>
> wrote:
>
> Hi Harsha,
>
> I mentioned policies and the authorizer. For example, with
> CreateTopicPolicy, you can implement the limits you describe. If
>
> you
>
> have
>
> ideas of how that should be improved, please submit a KIP. My
>
> point is
>
> that
>
> this KIP is not introducing any new functionality with regards to
>
> what
>
> rogue clients can do. It's using the existing protocol that is
>
> already
>
> exposed via the AdminClient. So, I don't think we need to address
>
> it in
>
> this KIP. Does that make sense?
>
> Ismael
>
> On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
>
> kafka@ harsha. io ( kafka@harsha.io ) >
>
> wrote:
>
> Ismael,
> Sure AdminClient can do that and we should've shipped a config or
>
> use
>
> the
>
> existing one to block that. Not all users are yet to upgrade to
>
> AdminClient
>
> and start using that to cause issues yet. In shared environment we
>
> should
>
> allow server to set sane defaults and not allow every client to go
>
> ahead
>
> create random no.of topic/partitions and replication factor. Even
>
> if
>
> the
>
> users want to allow topic creation proposed in the KIP , it makes
>
> sense to
>
> have some guards against the no.of partitions and replication
>
> factor.
>
> Authorizer is not always an answer to block requests and having
>
> users
>
> set
>
> server side configs to protect a multi-tenant environment is
>
> required.
>
> In a
>
> non-secure environment Authorizer is a blunt instrument either you
>
> end
>
> up
>
> blocking everyone or allowing everyone.
> I am asking to have server side that allow clients to create
>
> topics or
>
> not
>
> , if they are allowed set a ceiling on max no.of partitions and
> replication-factor.
>
> -Harsha
>
> On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com ( ismaelj@gmail.com
> )
>
> wrote:
>
> Harsha,
>
> Rogue clients can use the admin client to create topics and
>
> partitions.
>
> ACLs and policies can help in that case as well as this one.
>
> Ismael
>
> On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha. io (
> kafka@harsha.io ) >
>
> wrote:
>
> Hi Justine,
> Thanks for the KIP.
> "When server-side auto-creation is disabled, client-side
>
> auto-creation
>
> will try to use client-side configurations"
> If I understand correctly, this KIP is removing any server-side
>
> blocking
>
> client auto creation of topic?
> if so this will present potential issue of rogue client creating
>
> ton of
>
> topic-partitions and potentially bringing down the service for
>
> everyone
>
> or
>
> degrade the service itself.
> By reading the KIP its not clear to me that there is a clear way to
>
> block
>
> auto creation topics of all together from clients by server side
>
> config.
>
> Server side configs of default topic, partitions should take higher
> precedence and client shouldn't be able to create a topic with
>
> higher
>
> no.of
>
> partitions, replication than what server config specifies.
>
> Thanks,
> Harsha
>
> On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Hi all,
> I made some changes to the KIP. Hopefully this configuration change
>
> will
>
> make things a little clearer.
>
> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ ( https://cwiki.
> apache.org/confluence/display/KAFKA/ )
> KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
>
> Please let me know if you have any feedback or questions!
>
> Thank you,
> Justine
>
> On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache. org (
> cmccabe@apache.org ) >
>
> wrote:
>
> Hi Mickael,
>
> I think you bring up a good point. It would be better if we didn't
>
> ever
>
> have to set up client-side configuration for this feature, and
>
> KIP-464
>
> would let us skip this entirely.
>
> On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
>
> Hi Mickael,
> I agree that KIP-464 works on newer brokers, but I was a bit
>
> worried
>
> how
>
> things would play out on older brokers that* do not *have KIP 464
>
> included.
>
> Is it enough to throw an error in this case when producer configs
>
> are
>
> not
>
> specified?
>
> I think the right thing to do would be to log an error message in
>
> the
>
> client. We will need to have that capability in any case, to cover
> scenarios like the client trying to auto-create a topic that they
>
> don't
>
> have permission to create. Or a client trying to create a topic on
>
> a
>
> broker
>
> so old that CreateTopicsRequest is not supported.
>
> The big downside to relying on KIP-464 is that it is a very recent
>
> feature
>
> -- so recent that it hasn't even made its way to any official
>
> Apache
>
> release. It's scheduled for the upcoming 2.4 release in a few
>
> months.
>
> So if you view this KIP as a step towards removing broker-side
> auto-create, you might want to support older brokers just to
>
> accelerate
>
> adoption, and hasten the day when we can finally flip broker-side
> auto-create to off (or even remove it entirely).
>
> I have to agree, though, that having client-side configurations for
>
> number
>
> of partitions and replication factor is messy. Maybe it would be
>
> worth
>
> it
>
> to restrict support to post-KIP-464 brokers, if we could avoid
>
> creating
>
> more configs.
>
> best,
> Colin
>
> On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
>
> mickael. maison@ gmail. com ( mickael.maison@gmail.com )
>
> wrote:
>
> Hi Justine,
>
> We can rely on KIP-464 which allows to omit the partition count or
> replication factor when creating a topic. In that case, the broker
>
> defaults
>
> are used.
>
> On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Michael,
> That makes sense to me!
> To clarify, in the current state of the KIP, the producer does not
>
> rely
>
> on
>
> the broker to autocreate--if the broker's config is disabled, then
>
> the
>
> producer can autocreate on its own with a create topic request (the
>
> same
>
> type of request the admin client uses).
> However, if both configs are enabled, the broker will autocreate
>
> through
>
> a
>
> metadata request before the producer gets a chance. Of course, the
>
> way
>
> to
>
> avoid this, is to do as you suggested, and set
>
> the
>
> "allow_auto_topic_creation" field to false.
>
> I think the only thing we need to be careful with in this setup is
>
> without
>
> KIP 464, we can not use broker defaults for this topic. A user
>
> needs
>
> to
>
> specify the number of partition and replication factor in the
>
> config.
>
> An
>
> alternative to this is to have coded defaults for when these
>
> configs
>
> are
>
> unspecified, but it is not immediately apparent what these defaults
>
> should
>
> be.
>
> Thanks again for reading my KIP,
> Justine
>
> On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
>
> mickael. maison@ gmail. com ( mickael.maison@gmail.com )
>
> wrote:
>
> Hi Justine,
>
> Thanks for the response!
> In my opinion, it would be better if the producer did not rely at
>
> all
>
> on the broker auto create feature as this is what we're aiming to
> deprecate. When requesting metadata we can set the
> "allow_auto_topic_creation" field to false to avoid the broker auto
> creation. Then if the topic is not existing, send a
>
> CreateTopicRequest.
>
> What do you think?
>
> On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Currently the way it is implemented, the broker auto-creation
>
> configuration
>
> takes precedence. The producer will not use the CreateTopics
>
> request.
>
> (Technically it can--but the topic will already be created
>
> through
>
> the
>
> broker, so it will never try to create the topic.) It is possible
>
> to
>
> change this however, and I'd be happy to
>
> discuss
>
> the
>
> benefits of this alternative.
>
> Thank you,
> Justine
>
> On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
>
> mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP!
>
> In case auto creation is enabled on both the client and server,
>
> will
>
> the producer still use the AdminClient (CreateTopics request)
>
> to
>
> create topics? and not rely on the broker auto create. I'm
>
> guessing the
>
> answer is yes but can you make it explicit.
>
> Thank you
>
> On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Hi,
> Just a friendly reminder to take a look at this KIP if you
>
> have
>
> the
>
> time.
>
> I was thinking about broker vs. client default precedence,
>
> and I
>
> think it
>
> makes sense to keep the broker as the default used when both
>
> client-side
>
> and broker-side defaults are configured. The idea is that
>
> there
>
> would be
>
> pretty clear documentation, and that many systems with
>
> configurations
>
> that
>
> the client could not change would likely have the auto-create
>
> default
>
> off.
>
> (In cloud for example).
>
> It also seems like in most cases, the consumer config
> ' allow. auto. create. topics ( http://allow.auto.create.topics/ ) ' was
> created to actually prevent
>
> the
>
> creation
>
> of
>
> topics, so the loss of creation functionality will not be a
>
> big
>
> problem.
>
> I'm happy to discuss any other compatibility problems or
>
> components
>
> of
>
> this KIP.
>
> Thank you,
> Justine
>
> On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io )
>
> wrote:
>
> Hello all,
>
> I was looking at this KIP again, and there is a decision I
>
> made
>
> that I
>
> think is worth discussing.
>
> In the case where both the broker and producer's
> 'auto.create.topics.enable' are set to true, we have to
>
> choose
>
> either
>
> the
>
> broker configs or the producer configs for the replication
> factor/partitions.
>
> Currently, the decision is to have the broker defaults take
>
> precedence.
>
> (It is easier to do this in the implementation.) It also
>
> makes
>
> some
>
> sense
>
> for this behavior to take precedence since this behavior
>
> already
>
> occurs as
>
> the default.
>
> However, I was wondering if it would be odd for those who
>
> can
>
> only
>
> see
>
> the
>
> producer side to set configs for replication
>
> factor/partitions
>
> and
>
> see
>
> different results. Currently the documentation for the
>
> config
>
> states
>
> that
>
> the config values are only used when the broker config is
>
> not
>
> enabled,
>
> but
>
> this might not always be clear to the user. Changing the
>
> code
>
> to
>
> have
>
> the
>
> producer's configurations take precedence is possible, but
>
> I
>
> was
>
> wondering
>
> what everyone thought.
>
> Thank you,
> Justine
>
> On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Just a quick update--
>
> It seems that enabling both the broker and producer
>
> configs
>
> works
>
> fine,
>
> except that the broker configurations for partitions,
>
> replication
>
> factor
>
> take precedence.
> I don't know if that is something we would want to
>
> change, but
>
> I'll be
>
> updating the KIP for now to reflect this. Perhaps we would
>
> want to
>
> add more
>
> to the documentation of the the producer configs to
>
> clarify.
>
> Thank you,
> Justine
>
> On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Hi Colin,
>
> Thanks for looking at the KIP. I can definitely add to
>
> the
>
> title
>
> to
>
> make
>
> it more clear.
>
> It makes sense that both configurations could be turned
>
> on
>
> since
>
> there
>
> are many cases where the user can not control the
>
> server-side
>
> configurations. I was a little concerned about how both
>
> interacting
>
> would
>
> work out -- if there would be to many requests for new
>
> topics,
>
> for
>
> example.
>
> But it since it does make sense to allow both
>
> configurations
>
> enabled, I
>
> will test out some scenarios and I'll change the KIP to
>
> support
>
> this.
>
> I also agree with documentation about distinguishing the
>
> differences. I
>
> was having some trouble with the wording but I like the
>
> phrases
>
> "server-side" and "client-side." That's a good
>
> distinction I
>
> can
>
> use
>
> when
>
> describing.
>
> I'll try to update the KIP soon keeping everyone's input
>
> in
>
> mind.
>
> Thanks,
> Justine
>
> On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
>
> cmccabe@ apache. org ( cmccabe@apache.org )
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP. This seems like a good step towards
>
> removing
>
> server-side topic auto-creation.
>
> We should add included "client-side" to the title of
>
> the KIP
>
> somewhere,
>
> to make it clear that we're talking about client-side
>
> auto
>
> creation.
>
> The KIP says:
>
> In order to automatically create topics with the
>
> producer, the
>
> producer's
>
> auto.create.topics.enable config must be set to true
>
> and
>
> the
>
> broker
>
> config should be set to false
>
> From a user's point of view, this seems
>
> counter-intuitive.
>
> In
>
> order to
>
> auto-create topics the broker's
>
> auto.create.topics.enable
>
> config
>
> should be
>
> set to false? It seems like the server-side
>
> auto-create is
>
> unrelated to
>
> the client-side auto-create. We could have both turned
>
> on
>
> (and
>
> I'm
>
> sure
>
> that in the real world, people will try this
>
> configuration...)
>
> There's no
>
> reason not to support this, I think.
>
> We should add some documentation explaining the
>
> difference
>
> between
>
> server-side and client-side auto-creation. Without
>
> documentation,
>
> an admin
>
> might think that they had disabled all forms of
>
> auto-creation by
>
> setting
>
> the -side setting to false-- but this is not the case,
>
> of
>
> course.
>
> best,
> Colin
>
> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
>
> Hi Dhruvil,
>
> Thanks for reading the KIP!
> That was the general idea for deprecation. We would
>
> log a
>
> warning
>
> when the
>
> config is enabled on the broker.
> I also believe that there would be a change to
>
> documentation.
>
> If there is anything else that should be done, please
>
> let
>
> me
>
> know!
>
> Justine
>
> On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
>
> dhruvil@ confluent. io ( dhruvil@confluent.io ) >
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP, this is great!
>
> Could you add some more information about what
>
> deprecating
>
> the
>
> broker
>
> configuration means? Would we log a warning in the
>
> logs
>
> when
>
> auto
>
> topic
>
> creation is enabled on the broker, for example?
>
> Thanks,
> Dhruvil
>
> On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
>
> jolshan@ confluent. io ( jolshan@confluent.io ) >
>
> wrote:
>
> Hello all,
>
> I'd like to start a discussion thread for KIP-487. This KIP plans
>
> to
>
> deprecate the current system of
>
> auto-creating
>
> topics
>
> through requests to the metadata and give the
>
> producer the
>
> ability to
>
> automatically create topics instead.
>
> More information can be found here:
>
> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ ( https://cwiki.
> apache.org/confluence/display/KAFKA/ )
> KIP-487%3A+Automatic+Topic+Creation+on+Producer
>
> Thank you,
> Justine Olshan
>
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Colin McCabe <co...@cmccabe.xyz>.

On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmccabe@apache.org > wrote:
> 
> > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> >> 
> >> Hi Colin,
> >> "Hmm... I'm not sure I follow. Users don't have to build their own
> >> tooling, right? They can use any of the shell scripts that we've shipped
> >> in the last few releases. For example, if any of your users run it, this
> >> shell script will delete all of the topics from your non-security-enabled
> >> cluster:
> >> 
> >> 
> >> 
> >> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server
> >> localhost:9092 --list 2>/dev/null
> >> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server
> >> localhost:9092 --delete
> >> --topic
> >> 
> >> 
> >> 
> >> They will need to fill in the correct bootstrap servers list, of course,
> >> not localhost. This deletion script will work on some pretty old brokers,
> >> even back to the 0.10 releases. It seems a little odd to trust your users
> >> with this power, but not trust them to avoid changing a particular
> >> configuration key."
> >> 
> >> 
> >> 
> >> 
> >> The above will blocked by the server if we set delete.topic.enable to
> >> false and thats exactly what I am asking for.
> >> 
> >> 
> >> 
> > 
> > 
> > 
> > Hi Harsha,
> > 
> > 
> > 
> > 
> > I was wondering if someone was going to bring up that configuration :)
> > 
> > 
> > 
> > 
> > it's an interesting complication, but globally disabling topic deletion is
> > not very practical for most use-cases.
> > 
> > 
> > 
> > 
> > In any case, there are plenty of other bad things that users with full
> > permissions can do that aren't blocked by any server configuration. For
> > example, they can delete every record in every topic. I can write a script
> > for that too, and there's no server configuration you can set to disable
> > it. Or I could simply create hundreds of thousands of topics, until
> > cluster performance becomes unacceptable (this will be even more of a
> > problem if someone configured delete.topic.enable as false). Or publish
> > bad data to every topic, etc. etc.
> > 
> > 
> > 
> > 
> > The point I'm trying to make here is that you can't rely on these kind of
> > server-side configurations for security. At most, they're a way to set up
> > certain very simple policies. But the policies are so simple that they're
> > hardly ever useful any more.
> > 
> > 
> > 
> > 
> > For example, if the problem you want to solve is that you want a user to
> > only be able to create 50 topics and not delete anyone else's topics, you
> > can solve that with a CreateTopicsPolicy that limits the number of topics,
> > and some ACLs. There's no combination of auto.create.topics.enable and
> > delete.topic.enable that will help here.
> > 
> > 
> > 
> > 
> 
> Hi Colin,
> 
>             Well you gave the example that a user can delete the topics 
> just by running that script  :).  
> 
> I understand there are open APIs in Kafka and can lead to rogue clients 
> taking advantage of it without proper security in place.
> 
> What I am asking so far in this thread is , this KIP is changing the 
> producer behavior and its not backward compatible. 

The change is backwards compatible.  The default will still be server-side topic auto-creation, just like now.

You will have to specifically change the producer config to get the new behavior.

> We can still achieve 
> the main goal of the KIP which is to change MetadataRequest  creating 
> topics and send a CreateTopicRequest from Producer and also keep the 
> server side config to have precedence.  This KIP originally written to 
> have server side preference and there is not much explanation why it 
> changed to have Producer side preference.
> 
> Arguing that AdminClient can do that and so we are going to make 
> Producer do the same doesn't make sense.
> 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >> 
> >> 
> >> "The downside is that if we wanted to check a server side configuration
> >> before sending the create topics request, the code would be more complex.
> >> The behavior would also not be consistent with how topic auto-creation is
> >> handled in Kafka Streams."
> >> 
> >> 
> >> 
> >> 
> >> I am not sure why you need to check server side configuration before
> >> sending create topics request. A user enables producer side config to
> >> create topics.
> >> Producer sends a request to the broker and if the broker has
> >> auto.topic.create.enable to true (default) it will allow creation of
> >> topics. If it set to false it returns error back to the client.
> >> 
> >> 
> > 
> > 
> > 
> > auto.topic.create.enable has never affected CreateTopicsRequest. If you
> > submit a CreateTopicsRequest and you are authorized, a topic will be
> > created, regardless of what the value of auto.topic.create.enable is. This
> > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?)
> > 
> > 
> > 
> >> 
> >> 
> >> I don't see how this behavior will be different in Kafka streams. By
> >> default server allows the topic creation and with this KIP, It will only
> >> allow creation of topic when both producer and server side are turned on.
> >> Its exactly the same behavior in KIP-361.
> >> 
> >> 
> >> 
> > 
> > 
> > 
> > I suggest running a streams job as a test. It creates the topics it needs
> > using AdminClient. The value of auto.topic.create.enable is not relevant.
> > Whether it is true or false, the topics will still be created.
> > 
> > 
> > 
> >> 
> >> 
> >> "In general, it would be nice if we could keep the security and access
> >> control model simple and not introduce a lot of special cases and
> >> exceptions. Kafka has basically converged on using ACLs and
> >> CreateTopicPolicy classes to control who can create topics and where.
> >> Adding more knobs seems like a step backwards, especially when the
> >> proposed knobs don't work consistently across components, and don't
> >> provide true security." This is not about access control at all. Shipping
> >> sane defaults should be prioritized.
> >> 
> >> 
> >> 
> > 
> > 
> > 
> > I don't think the default is really the question here. I think everyone
> > agrees that the default for client-side auto-topic creation should be off.
> > 
> > 
> > 
> > 
> > 
> 
> My point goes back to KIP-361 why we kept the server side config to 
> have preference. We can keep bringing back the discussion of security 
> policies but a producer client overiding server side setting is not 
> just security policy issue .
> 
> "Producer client can override server side setting and not consumer and 
> also producer can only override creation of topic but no the 
> replication and partitions. " This doesn't make sense to me and why we 
> want to introduce such a confusing behavior. 
> 
> > 
> > 
> > 
> > The scenarios that you're describing (such as dealing with a poorly
> > configured client that tries to create topics it should not) do seem to be
> > about access control.
> > 
> > 
> > 
> >> 
> >> 
> >> We keep talking about CreateTopicPolicy and yet we don't have default one
> >> and asking every user of Kafka implement their own doesn't make sense
> >> here.
> >> 
> >> 
> >> 
> > 
> > 
> > 
> > The point of CreateTopicPolicy is exactly that you can implement your own,
> > though. It's a way of customizing what the policy is in your specific
> > cluster.
> > 
> > 
> > 
> > 
> > I agree that most people don't want to write a custom policy. But most of
> > those people are satisfied with just setting up ACLs to set up simple
> > policies like this user can create topics, that user can't, etc. That's
> > why I suggested adding support for ACLs to insecure clusters might be
> > helpful.
> > 
> > 
> > 
> > 
> > Also, just as a side note, wouldn't it would be impossible for us to
> > specify a default CreateTopicsPolicy that did anything at this point
> > anyway without breaking backwards compatibility? Maybe I'm
> > misunderstanding the proposal.
> > 
> > 
> > 
> >> 
> >> 
> >> I am asking about exact behavior that we shipped for consumer side https:/
> >> / cwiki. apache. org/ confluence/ display/ KAFKA/ KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> >> (
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> >> )
> >> 
> >> 
> >> 
> >> 
> >> I still didn't get any response why this behavior shouldn't be exactly
> >> like Kafka consumer and why do we want to have producer to overrider
> >> server side config and while not allowing consumer to do so.
> >> We are not even allowing the same contract and this will cause more
> >> confusion from the users standpoint.
> >> 
> >> 
> > 
> > 
> > 
> > Hmm. The consumer already has a way to override the server side
> > configuration. See KIP-361: Add Consumer Configuration to Disable Auto
> > Topic Creation. This lets the consumer skip auto-creating topics, even if
> > auto-creation is enabled on the broker.
> > 
> > 
> > 
> > 
> 
> Yes that's what I am saying and it doesn't allow topic creation if the 
> server side auto-creation is disabled and in consumer its enabled.   I 
> would like to see the exact behavior for producer as stated in KIP-361.
> 
> > 
> > 
> > 
> > To be fair, the KIP should probably discuss why we don't want to implement
> > client-side topic creation in the consumer in "rejected alternatives."
> > Maybe Justine can add more context here in the KIP.
> > 
> > 
> > 
> > 
> > The last time we talked about this, the reasoning was that we wanted to
> > eventually get rid of consumer-side auto-topic creation entirely, and so
> > it wasn't worth implementing an improved way of doing it. Producer side
> > auto topic-creation, in contrast, will probably stick around, although
> > we'd like to deprecate and remove the problematic server-side mechanism
> > over time.
> > 
> > 
> > 
> > 
> 
> We should implement the producer side topic creation with proper create 
> topic request  and I am in favor of this KIP.  
> 
> All I am asking is don't make the producer to override server side 
> config and keep it similar to KIP-361 just like consumer, it honors 
> what set on server side which takes the precedence.   Changing this 
> behavior in producer is backward incompatible and will catch users by 
> surprise. 
> 
> All arguments for enforcing producer side overriding config can still 
> be achieved. Server side auto creation turned on by default and  any 
> new producer client setting their auto creation config to true will 
> create topics in default behavior and any user who set server side to 
> false and upgrades to latest kafka with this KIP will not see any 
> unwanted behavior from clients.  IMO this is not a unreasonable request 
> on this KIP and this requested behavior is exactly what the KIP 
> initially proposed.
> 
> Thanks,
> 
> Harsha
> 
> > 
> > 
> > 
> > best,
> > Colin
> > 
> > 
> >> 
> >> 
> >> On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache. org (
> >> cmccabe@apache.org ) > wrote:
> >> 
> >> 
> >> 
> >>> 
> >>> 
> >>> On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>> 
> >>>> Not sure how the AdminClient applies here, It is an external API and is
> >>>> not part of KafkaProducer so any user who updates to latest version of
> >>>> Kafka with this feature get to create the topics.
> >>>> They have to build a tooling around AdminClient allow themselves to
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> create
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>> 
> >>>> topics.
> >>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> Hi Harsha,
> >>> 
> >>> 
> >>> 
> >>> 
> >>> Hmm... I'm not sure I follow. Users don't have to build their own tooling,
> >>> right? They can use any of the shell scripts that we've shipped in the
> >>> last few releases. For example, if any of your users run it, this shell
> >>> script will delete all of the topics from your non-security-enabled
> >>> cluster:
> >>> 
> >>> 
> >>> 
> >>> 
> >>> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server
> >>> localhost:9092 --list 2>/dev/null
> >>> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server
> >>> localhost:9092 --delete
> >>> --topic
> >>> 
> >>> 
> >>> 
> >>> They will need to fill in the correct bootstrap servers list, of course,
> >>> not localhost. This deletion script will work on some pretty old brokers,
> >>> even back to the 0.10 releases. It seems a little odd to trust your users
> >>> with this power, but not trust them to avoid changing a particular
> >>> configuration key.
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>> 
> >>>> There is no behavior in Kafka producer that allowed users to delete the
> >>>> topics or delete the records. So citing them as an example doesn't makes
> >>>> sense in this context.
> >>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> I think Kafka Streams is relevant here. After all, it's software that we
> >>> ship as part of the official Kafka release. And it auto-creates topics
> >>> even when auto.create.topics.enable is set to false on the broker.
> >>> 
> >>> 
> >>> 
> >>> 
> >>> I think that auto.create.topics.enable was never intended as a security
> >>> setting to control access. It was intended as a way to disable the
> >>> broker-side auto-create feature specifically, because there were some
> >>> problems with that specific feature. Broker-side auto-creation is
> >>> frustrating because it's on by default, and because it can auto-create
> >>> topics even if the producer or consumer didn't explicitly ask for that.
> >>> Neither of those problems applies to this KIP: producers have to
> >>> specifically opt in, and it won't be on by default. Basically, we think
> >>> that client-side auto-creation is actually a lot better-- hence this KIP
> >>> :)
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>> 
> >>>> But there is
> >>>> a functionality which allowed creation of topics if they don't exist in
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> the
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>> 
> >>>> cluster and this behavior could be controlled by a config on the server
> >>>> side. Now with this KIP we are allowing producer to make that decision
> >>>> without any gateway on the server via configs. Any user who is not aware
> >>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> of
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>> 
> >>>> such changes
> >>>> can accidentally create these topics and we are essentially removing a
> >>>> config that exists in brokers today to block this accidental creation and
> >>>> allowing clients to take control.
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> Again, I hope I'm not misinterpreting, but I don't see how this can be
> >>> turned on accidentally. The user would have to specifically turn this on
> >>> in the producer by setting the configuration key.
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>> 
> >>>> I still didn't get any positives of not having server side configs? if you
> >>>> want to turn them on and allow any client to create topics set the default
> >>>> to true
> >>>> and allow users who doesn't want to have this feature let them turn them
> >>>> off. It will be the exact behavior as it is today , as far as producer is
> >>>> concerned. I am not
> >>>> understanding why not having server side configs to gateways such a hard
> >>>> requirement and this behavior exists today. As far I am concerned this
> >>>> breaks the backward compatibility.
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> The downside is that if we wanted to check a server side configuration
> >>> before sending the create topics request, the code would be more complex.
> >>> The behavior would also not be consistent with how topic auto-creation is
> >>> handled in Kafka Streams.
> >>> 
> >>> 
> >>> 
> >>> 
> >>> In general, it would be nice if we could keep the security and access
> >>> control model simple and not introduce a lot of special cases and
> >>> exceptions. Kafka has basically converged on using ACLs and
> >>> CreateTopicPolicy classes to control who can create topics and where.
> >>> Adding more knobs seems like a step backwards, especially when the
> >>> proposed knobs don't work consistently across components, and don't
> >>> provide true security.
> >>> 
> >>> 
> >>> 
> >>> 
> >>> Maybe we should support an insecure mode where there are still users and
> >>> ACLs. That would help people who don't want to set up Kerberos, etc. but
> >>> still want to protect against misconfigured clients or accidents. Hadoop
> >>> has something like this, and I think it was useful. NFS also supported
> >>> (supports?) a mode where you just pass whatever user ID you want and the
> >>> system believes you. These things clearly don't protect against malicious
> >>> users, but they can help set up policies when needed.
> >>> 
> >>> 
> >>> 
> >>> best,
> >>> Colin
> >>> 
> >>> 
> >>>> 
> >>>> 
> >>>> Thanks,
> >>>> Harsha
> >>>> 
> >>>> 
> >>>> 
> >>>> On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache. org (
> >>>> cmccabe@apache.org ) > wrote:
> >>>> 
> >>>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> Hi Harsha,
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> Thanks for taking a look.
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> I'm not sure I follow the discussion about AdminClient.
> >>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> KafkaAdminClient
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> has been around for about 2 years at this point as a public class.
> >>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> There
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> are many programs that use it to automatically create topics. Kafka
> >>>>> Streams does this, for example. If any of your users run Kafka
> >>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> Streams,
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> they will be auto-creating topics, regardless of what setting you use
> >>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> for
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> auto.create.topics.enable.
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> A big part of the annoyance of auto-topic creation right now is that
> >>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> it is
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> on by default. The new configuration proposed by KIP-487 wouldn't be.
> >>>>> Users would have to explicitly opt in to the new behavior of
> >>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> client-side
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> topic creation. If you run without security, you're already putting a
> >>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> huge
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> amount of trust in your users. For example, you trust them not to
> >>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> delete
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> records with the kafka-delete-records. sh ( http://kafka-delete-records.sh/
> >>>>> ) command, or delete topics with kafka-topics. sh ( http://kafka-topics.sh/
> >>>>> ). Trusting them not to set a certain config value seems minor in
> >>>>> comparison, right?
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> best,
> >>>>> Colin
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> Hi,
> >>>>>> Even with policies one needs to implement that, so for every
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> user who
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> doesn't want a producer to create topics or have limits around
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> partitions &
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> replication factor they have to implement a policy. The KIP is changing
> >>>>>> the behavior , it might not be introducing
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> the
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> new functionality but it will enable producers to override the create
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> topic
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> config settings on the broker. What I am asking for to provide a
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> config
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> that will disable auto creation of topics and if its enabled set some
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> sane
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> defaults so that clients can create a topic with in those limits. I
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> don't
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> see how this not related to this KIP.
> >>>>>> If the server config options as I suggested doesn't interest
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> you at
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> least have a default CreateTopicPolices in place.
> >>>>>> To give an example, In our environment we disable the
> >>>>>> auto.create.topic.enable and force users to go through a centralized
> >>>>>> service as we want capture more details about the topic creation and
> >>>>>> requirements. With this KIP, a producer can create a topic with no
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> bounds.
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> Another example max.message.size we define that at cluster level
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> and one
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> can override max.messsage.size at topic level, any misconfiguration
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> at
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> this
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> will cause service degradation. Its not always about the rogue
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> clients,
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> users can easily misconfigure and can cause an outage. Again we can talk
> >>>>>> about CreateTopicPolicy but without having a
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> default
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> implementation and asking everyone to implement their own while
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> changing
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> the behavior in producer doesn't make sense to me.
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> Thanks,
> >>>>>> Harsha
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me. uk (
> >>>>>> ismael@juma.me.uk ) >
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> wrote:
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Harsha,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I mentioned policies and the authorizer. For example, with
> >>>>>>> CreateTopicPolicy, you can implement the limits you describe. If
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> you
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> have
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> ideas of how that should be improved, please submit a KIP. My
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> point is
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> that
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> this KIP is not introducing any new functionality with regards to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> what
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> rogue clients can do. It's using the existing protocol that is
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> already
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> exposed via the AdminClient. So, I don't think we need to address
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> it in
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> this KIP. Does that make sense?
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Ismael
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> kafka@ harsha. io ( kafka@harsha.io ) >
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Ismael,
> >>>>>>> Sure AdminClient can do that and we should've shipped a config or
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> use
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> the
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> existing one to block that. Not all users are yet to upgrade to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> AdminClient
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> and start using that to cause issues yet. In shared environment we
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> should
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> allow server to set sane defaults and not allow every client to go
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> ahead
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> create random no.of topic/partitions and replication factor. Even
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> if
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> the
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> users want to allow topic creation proposed in the KIP , it makes
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> sense to
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> have some guards against the no.of partitions and replication
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> factor.
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Authorizer is not always an answer to block requests and having
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> users
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> set
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> server side configs to protect a multi-tenant environment is
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> required.
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> In a
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> non-secure environment Authorizer is a blunt instrument either you
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> end
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> up
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> blocking everyone or allowing everyone.
> >>>>>>> I am asking to have server side that allow clients to create
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> topics or
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> 
> >>>>> not
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> , if they are allowed set a ceiling on max no.of partitions and
> >>>>>>> replication-factor.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> -Harsha
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com ( ismaelj@gmail.com )
> >>>>>>> > wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Harsha,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Rogue clients can use the admin client to create topics and
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> partitions.
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> ACLs and policies can help in that case as well as this one.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Ismael
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha. io (
> >>>>>>> kafka@harsha.io ) >
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Justine,
> >>>>>>> Thanks for the KIP.
> >>>>>>> "When server-side auto-creation is disabled, client-side
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> auto-creation
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> will try to use client-side configurations"
> >>>>>>> If I understand correctly, this KIP is removing any server-side
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> blocking
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> client auto creation of topic?
> >>>>>>> if so this will present potential issue of rogue client creating
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> ton of
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> topic-partitions and potentially bringing down the service for
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> everyone
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> or
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> degrade the service itself.
> >>>>>>> By reading the KIP its not clear to me that there is a clear way to
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> block
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> auto creation topics of all together from clients by server side
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> config.
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Server side configs of default topic, partitions should take higher
> >>>>>>> precedence and client shouldn't be able to create a topic with
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> higher
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> no.of
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> partitions, replication than what server config specifies.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks,
> >>>>>>> Harsha
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> jolshan@ confluent. io ( jolshan@confluent.io ) >
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi all,
> >>>>>>> I made some changes to the KIP. Hopefully this configuration change
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> will
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> make things a little clearer.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/ ) KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Please let me know if you have any feedback or questions!
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thank you,
> >>>>>>> Justine
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache. org (
> >>>>>>> cmccabe@apache.org ) >
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Mickael,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I think you bring up a good point. It would be better if we didn't
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> ever
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> have to set up client-side configuration for this feature, and
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> KIP-464
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> would let us skip this entirely.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Mickael,
> >>>>>>> I agree that KIP-464 works on newer brokers, but I was a bit
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> worried
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> how
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> things would play out on older brokers that* do not *have KIP 464
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> included.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Is it enough to throw an error in this case when producer configs
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> are
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> not
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> specified?
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I think the right thing to do would be to log an error message in
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> the
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> client. We will need to have that capability in any case, to cover
> >>>>>>> scenarios like the client trying to auto-create a topic that they
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> don't
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> have permission to create. Or a client trying to create a topic on
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> a
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> broker
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> so old that CreateTopicsRequest is not supported.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> The big downside to relying on KIP-464 is that it is a very recent
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> feature
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> -- so recent that it hasn't even made its way to any official
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> Apache
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> release. It's scheduled for the upcoming 2.4 release in a few
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> months.
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> So if you view this KIP as a step towards removing broker-side
> >>>>>>> auto-create, you might want to support older brokers just to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> accelerate
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> adoption, and hasten the day when we can finally flip broker-side
> >>>>>>> auto-create to off (or even remove it entirely).
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I have to agree, though, that having client-side configurations for
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> number
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> of partitions and replication factor is messy. Maybe it would be
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> worth
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> it
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> to restrict support to post-KIP-464 brokers, if we could avoid
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> creating
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> more configs.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> best,
> >>>>>>> Colin
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Justine,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> We can rely on KIP-464 which allows to omit the partition count or
> >>>>>>> replication factor when creating a topic. In that case, the broker
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> defaults
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> are used.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> jolshan@ confluent. io ( jolshan@confluent.io ) >
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Michael,
> >>>>>>> That makes sense to me!
> >>>>>>> To clarify, in the current state of the KIP, the producer does not
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> rely
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> on
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the broker to autocreate--if the broker's config is disabled, then
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> producer can autocreate on its own with a create topic request (the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> same
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> type of request the admin client uses).
> >>>>>>> However, if both configs are enabled, the broker will autocreate
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> through
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> a
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> metadata request before the producer gets a chance. Of course, the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> way
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> avoid this, is to do as you suggested, and set
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> "allow_auto_topic_creation" field to false.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I think the only thing we need to be careful with in this setup is
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> without
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> KIP 464, we can not use broker defaults for this topic. A user
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> needs
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> specify the number of partition and replication factor in the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> config.
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> An
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> alternative to this is to have coded defaults for when these
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> configs
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> are
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> unspecified, but it is not immediately apparent what these defaults
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> should
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> be.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks again for reading my KIP,
> >>>>>>> Justine
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> mickael. maison@ gmail. com ( mickael.maison@gmail.com )
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Justine,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks for the response!
> >>>>>>> In my opinion, it would be better if the producer did not rely at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> all
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> on the broker auto create feature as this is what we're aiming to
> >>>>>>> deprecate. When requesting metadata we can set the
> >>>>>>> "allow_auto_topic_creation" field to false to avoid the broker auto
> >>>>>>> creation. Then if the topic is not existing, send a
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> CreateTopicRequest.
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> What do you think?
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Currently the way it is implemented, the broker auto-creation
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> configuration
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> takes precedence. The producer will not use the CreateTopics
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> request.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> (Technically it can--but the topic will already be created
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> through
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> broker, so it will never try to create the topic.) It is possible
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> to
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> change this however, and I'd be happy to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> discuss
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> benefits of this alternative.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thank you,
> >>>>>>> Justine
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Justine,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks for the KIP!
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> In case auto creation is enabled on both the client and server,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> will
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the producer still use the AdminClient (CreateTopics request)
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> create topics? and not rely on the broker auto create. I'm
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> guessing the
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> answer is yes but can you make it explicit.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thank you
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi,
> >>>>>>> Just a friendly reminder to take a look at this KIP if you
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> have
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> time.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I was thinking about broker vs. client default precedence,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> and I
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> think it
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> makes sense to keep the broker as the default used when both
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> client-side
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> and broker-side defaults are configured. The idea is that
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> there
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> would be
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> pretty clear documentation, and that many systems with
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> configurations
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> that
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the client could not change would likely have the auto-create
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> default
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> off.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> (In cloud for example).
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> It also seems like in most cases, the consumer config
> >>>>>>> ' allow. auto. create. topics ( http://allow.auto.create.topics/ ) ' was
> >>>>>>> created to actually prevent
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> creation
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> of
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> topics, so the loss of creation functionality will not be a
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> big
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> problem.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I'm happy to discuss any other compatibility problems or
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> components
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> of
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> this KIP.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thank you,
> >>>>>>> Justine
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> jolshan@ confluent. io ( jolshan@confluent.io )
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hello all,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I was looking at this KIP again, and there is a decision I
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> made
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> that I
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> think is worth discussing.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> In the case where both the broker and producer's
> >>>>>>> 'auto.create.topics.enable' are set to true, we have to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> choose
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> either
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> broker configs or the producer configs for the replication
> >>>>>>> factor/partitions.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Currently, the decision is to have the broker defaults take
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> precedence.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> (It is easier to do this in the implementation.) It also
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> makes
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> some
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> sense
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> for this behavior to take precedence since this behavior
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> already
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> occurs as
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the default.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> However, I was wondering if it would be odd for those who
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> can
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> only
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> see
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> producer side to set configs for replication
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> factor/partitions
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> and
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> see
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> different results. Currently the documentation for the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> config
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> states
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> that
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the config values are only used when the broker config is
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> not
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> enabled,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> but
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> this might not always be clear to the user. Changing the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> code
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> have
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> producer's configurations take precedence is possible, but
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> was
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wondering
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> what everyone thought.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thank you,
> >>>>>>> Justine
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Just a quick update--
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> It seems that enabling both the broker and producer
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> configs
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> works
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> fine,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> except that the broker configurations for partitions,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> replication
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> factor
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> take precedence.
> >>>>>>> I don't know if that is something we would want to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> change, but
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I'll be
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> updating the KIP for now to reflect this. Perhaps we would
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> want to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> add more
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> to the documentation of the the producer configs to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> clarify.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thank you,
> >>>>>>> Justine
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Colin,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks for looking at the KIP. I can definitely add to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> title
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> make
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> it more clear.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> It makes sense that both configurations could be turned
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> on
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> since
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> there
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> are many cases where the user can not control the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> server-side
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> configurations. I was a little concerned about how both
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> interacting
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> would
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> work out -- if there would be to many requests for new
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> topics,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> for
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> example.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> But it since it does make sense to allow both
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> configurations
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> enabled, I
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> will test out some scenarios and I'll change the KIP to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> support
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> this.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I also agree with documentation about distinguishing the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> differences. I
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> was having some trouble with the wording but I like the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> phrases
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> "server-side" and "client-side." That's a good
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> distinction I
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> can
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> use
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> when
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> describing.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I'll try to update the KIP soon keeping everyone's input
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> in
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> mind.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks,
> >>>>>>> Justine
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> cmccabe@ apache. org ( cmccabe@apache.org )
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Justine,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks for the KIP. This seems like a good step towards
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> removing
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> server-side topic auto-creation.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> We should add included "client-side" to the title of
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the KIP
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> somewhere,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> to make it clear that we're talking about client-side
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> auto
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> creation.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> The KIP says:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> In order to automatically create topics with the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> producer, the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> producer's
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> auto.create.topics.enable config must be set to true
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> and
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> broker
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> config should be set to false
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> From a user's point of view, this seems
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> counter-intuitive.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> In
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> order to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> auto-create topics the broker's
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> auto.create.topics.enable
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> config
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> should be
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> set to false? It seems like the server-side
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> auto-create is
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> unrelated to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the client-side auto-create. We could have both turned
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> on
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> (and
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I'm
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> sure
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> that in the real world, people will try this
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> configuration...)
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> There's no
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> reason not to support this, I think.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> We should add some documentation explaining the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> difference
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> between
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> server-side and client-side auto-creation. Without
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> documentation,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> an admin
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> might think that they had disabled all forms of
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> auto-creation by
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> setting
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the -side setting to false-- but this is not the case,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> of
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> course.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> best,
> >>>>>>> Colin
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Dhruvil,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks for reading the KIP!
> >>>>>>> That was the general idea for deprecation. We would
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> log a
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> warning
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> when the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> config is enabled on the broker.
> >>>>>>> I also believe that there would be a change to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> documentation.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> If there is anything else that should be done, please
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> let
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> me
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> know!
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Justine
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> dhruvil@ confluent. io ( dhruvil@confluent.io ) >
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hi Justine,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks for the KIP, this is great!
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Could you add some more information about what
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> deprecating
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> broker
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> configuration means? Would we log a warning in the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> logs
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> when
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> auto
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> topic
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> creation is enabled on the broker, for example?
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thanks,
> >>>>>>> Dhruvil
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Hello all,
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I'd like to start a discussion thread for KIP-487. This KIP plans
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >>> to
> >>> 
> >>> 
> >>> 
> >>>> 
> >>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> deprecate the current system of
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> auto-creating
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> topics
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> through requests to the metadata and give the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> producer the
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> ability to
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> automatically create topics instead.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> More information can be found here:
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/ ) KIP-487%3A+Automatic+Topic+Creation+on+Producer
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Thank you,
> >>>>>>> Justine Olshan
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >> 
> >> 
> > 
> > 
> >

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Harsha Ch <ha...@gmail.com>.

On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmccabe@apache.org > wrote:

> 
> 
> 
> On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> 
> 
> 
>> 
>> 
>> Hi Colin,
>> "Hmm... I'm not sure I follow. Users don't have to build their own
>> tooling, right? They can use any of the shell scripts that we've shipped
>> in the last few releases. For example, if any of your users run it, this
>> shell script will delete all of the topics from your non-security-enabled
>> cluster:
>> 
>> 
>> 
>> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server
>> localhost:9092 --list 2>/dev/null
>> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server
>> localhost:9092 --delete
>> --topic
>> 
>> 
>> 
>> They will need to fill in the correct bootstrap servers list, of course,
>> not localhost. This deletion script will work on some pretty old brokers,
>> even back to the 0.10 releases. It seems a little odd to trust your users
>> with this power, but not trust them to avoid changing a particular
>> configuration key."
>> 
>> 
>> 
>> 
>> The above will blocked by the server if we set delete.topic.enable to
>> false and thats exactly what I am asking for.
>> 
>> 
>> 
> 
> 
> 
> Hi Harsha,
> 
> 
> 
> 
> I was wondering if someone was going to bring up that configuration :)
> 
> 
> 
> 
> it's an interesting complication, but globally disabling topic deletion is
> not very practical for most use-cases.
> 
> 
> 
> 
> In any case, there are plenty of other bad things that users with full
> permissions can do that aren't blocked by any server configuration. For
> example, they can delete every record in every topic. I can write a script
> for that too, and there's no server configuration you can set to disable
> it. Or I could simply create hundreds of thousands of topics, until
> cluster performance becomes unacceptable (this will be even more of a
> problem if someone configured delete.topic.enable as false). Or publish
> bad data to every topic, etc. etc.
> 
> 
> 
> 
> The point I'm trying to make here is that you can't rely on these kind of
> server-side configurations for security. At most, they're a way to set up
> certain very simple policies. But the policies are so simple that they're
> hardly ever useful any more.
> 
> 
> 
> 
> For example, if the problem you want to solve is that you want a user to
> only be able to create 50 topics and not delete anyone else's topics, you
> can solve that with a CreateTopicsPolicy that limits the number of topics,
> and some ACLs. There's no combination of auto.create.topics.enable and
> delete.topic.enable that will help here.
> 
> 
> 
> 

Hi Colin,

            Well you gave the example that a user can delete the topics just by running that script  :).  

I understand there are open APIs in Kafka and can lead to rogue clients taking advantage of it without proper security in place.

What I am asking so far in this thread is , this KIP is changing the producer behavior and its not backward compatible. We can still achieve the main goal of the KIP which is to change MetadataRequest  creating topics and send a CreateTopicRequest from Producer and also keep the server side config to have precedence.  This KIP originally written to have server side preference and there is not much explanation why it changed to have Producer side preference.

Arguing that AdminClient can do that and so we are going to make Producer do the same doesn't make sense.

> 
> 
> 
> 
> 
> 
> 
>> 
>> 
>> "The downside is that if we wanted to check a server side configuration
>> before sending the create topics request, the code would be more complex.
>> The behavior would also not be consistent with how topic auto-creation is
>> handled in Kafka Streams."
>> 
>> 
>> 
>> 
>> I am not sure why you need to check server side configuration before
>> sending create topics request. A user enables producer side config to
>> create topics.
>> Producer sends a request to the broker and if the broker has
>> auto.topic.create.enable to true (default) it will allow creation of
>> topics. If it set to false it returns error back to the client.
>> 
>> 
> 
> 
> 
> auto.topic.create.enable has never affected CreateTopicsRequest. If you
> submit a CreateTopicsRequest and you are authorized, a topic will be
> created, regardless of what the value of auto.topic.create.enable is. This
> behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?)
> 
> 
> 
>> 
>> 
>> I don't see how this behavior will be different in Kafka streams. By
>> default server allows the topic creation and with this KIP, It will only
>> allow creation of topic when both producer and server side are turned on.
>> Its exactly the same behavior in KIP-361.
>> 
>> 
>> 
> 
> 
> 
> I suggest running a streams job as a test. It creates the topics it needs
> using AdminClient. The value of auto.topic.create.enable is not relevant.
> Whether it is true or false, the topics will still be created.
> 
> 
> 
>> 
>> 
>> "In general, it would be nice if we could keep the security and access
>> control model simple and not introduce a lot of special cases and
>> exceptions. Kafka has basically converged on using ACLs and
>> CreateTopicPolicy classes to control who can create topics and where.
>> Adding more knobs seems like a step backwards, especially when the
>> proposed knobs don't work consistently across components, and don't
>> provide true security." This is not about access control at all. Shipping
>> sane defaults should be prioritized.
>> 
>> 
>> 
> 
> 
> 
> I don't think the default is really the question here. I think everyone
> agrees that the default for client-side auto-topic creation should be off.
> 
> 
> 
> 
> 

My point goes back to KIP-361 why we kept the server side config to have preference. We can keep bringing back the discussion of security policies but a producer client overiding server side setting is not just security policy issue .

"Producer client can override server side setting and not consumer and also producer can only override creation of topic but no the replication and partitions. " This doesn't make sense to me and why we want to introduce such a confusing behavior. 

> 
> 
> 
> The scenarios that you're describing (such as dealing with a poorly
> configured client that tries to create topics it should not) do seem to be
> about access control.
> 
> 
> 
>> 
>> 
>> We keep talking about CreateTopicPolicy and yet we don't have default one
>> and asking every user of Kafka implement their own doesn't make sense
>> here.
>> 
>> 
>> 
> 
> 
> 
> The point of CreateTopicPolicy is exactly that you can implement your own,
> though. It's a way of customizing what the policy is in your specific
> cluster.
> 
> 
> 
> 
> I agree that most people don't want to write a custom policy. But most of
> those people are satisfied with just setting up ACLs to set up simple
> policies like this user can create topics, that user can't, etc. That's
> why I suggested adding support for ACLs to insecure clusters might be
> helpful.
> 
> 
> 
> 
> Also, just as a side note, wouldn't it would be impossible for us to
> specify a default CreateTopicsPolicy that did anything at this point
> anyway without breaking backwards compatibility? Maybe I'm
> misunderstanding the proposal.
> 
> 
> 
>> 
>> 
>> I am asking about exact behavior that we shipped for consumer side https:/
>> / cwiki. apache. org/ confluence/ display/ KAFKA/ KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
>> (
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
>> )
>> 
>> 
>> 
>> 
>> I still didn't get any response why this behavior shouldn't be exactly
>> like Kafka consumer and why do we want to have producer to overrider
>> server side config and while not allowing consumer to do so.
>> We are not even allowing the same contract and this will cause more
>> confusion from the users standpoint.
>> 
>> 
> 
> 
> 
> Hmm. The consumer already has a way to override the server side
> configuration. See KIP-361: Add Consumer Configuration to Disable Auto
> Topic Creation. This lets the consumer skip auto-creating topics, even if
> auto-creation is enabled on the broker.
> 
> 
> 
> 

Yes that's what I am saying and it doesn't allow topic creation if the server side auto-creation is disabled and in consumer its enabled.   I would like to see the exact behavior for producer as stated in KIP-361.

> 
> 
> 
> To be fair, the KIP should probably discuss why we don't want to implement
> client-side topic creation in the consumer in "rejected alternatives."
> Maybe Justine can add more context here in the KIP.
> 
> 
> 
> 
> The last time we talked about this, the reasoning was that we wanted to
> eventually get rid of consumer-side auto-topic creation entirely, and so
> it wasn't worth implementing an improved way of doing it. Producer side
> auto topic-creation, in contrast, will probably stick around, although
> we'd like to deprecate and remove the problematic server-side mechanism
> over time.
> 
> 
> 
> 

We should implement the producer side topic creation with proper create topic request  and I am in favor of this KIP.  

All I am asking is don't make the producer to override server side config and keep it similar to KIP-361 just like consumer, it honors what set on server side which takes the precedence.   Changing this behavior in producer is backward incompatible and will catch users by surprise. 

All arguments for enforcing producer side overriding config can still be achieved. Server side auto creation turned on by default and  any new producer client setting their auto creation config to true will create topics in default behavior and any user who set server side to false and upgrades to latest kafka with this KIP will not see any unwanted behavior from clients.  IMO this is not a unreasonable request on this KIP and this requested behavior is exactly what the KIP initially proposed.

Thanks,

Harsha

> 
> 
> 
> best,
> Colin
> 
> 
>> 
>> 
>> On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache. org (
>> cmccabe@apache.org ) > wrote:
>> 
>> 
>> 
>>> 
>>> 
>>> On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
>>> 
>>> 
>>> 
>>>> 
>>>> 
>>>> Not sure how the AdminClient applies here, It is an external API and is
>>>> not part of KafkaProducer so any user who updates to latest version of
>>>> Kafka with this feature get to create the topics.
>>>> They have to build a tooling around AdminClient allow themselves to
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> create
>>> 
>>> 
>>> 
>>>> 
>>>> 
>>>> topics.
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> Hi Harsha,
>>> 
>>> 
>>> 
>>> 
>>> Hmm... I'm not sure I follow. Users don't have to build their own tooling,
>>> right? They can use any of the shell scripts that we've shipped in the
>>> last few releases. For example, if any of your users run it, this shell
>>> script will delete all of the topics from your non-security-enabled
>>> cluster:
>>> 
>>> 
>>> 
>>> 
>>> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server
>>> localhost:9092 --list 2>/dev/null
>>> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server
>>> localhost:9092 --delete
>>> --topic
>>> 
>>> 
>>> 
>>> They will need to fill in the correct bootstrap servers list, of course,
>>> not localhost. This deletion script will work on some pretty old brokers,
>>> even back to the 0.10 releases. It seems a little odd to trust your users
>>> with this power, but not trust them to avoid changing a particular
>>> configuration key.
>>> 
>>> 
>>> 
>>>> 
>>>> 
>>>> There is no behavior in Kafka producer that allowed users to delete the
>>>> topics or delete the records. So citing them as an example doesn't makes
>>>> sense in this context.
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> I think Kafka Streams is relevant here. After all, it's software that we
>>> ship as part of the official Kafka release. And it auto-creates topics
>>> even when auto.create.topics.enable is set to false on the broker.
>>> 
>>> 
>>> 
>>> 
>>> I think that auto.create.topics.enable was never intended as a security
>>> setting to control access. It was intended as a way to disable the
>>> broker-side auto-create feature specifically, because there were some
>>> problems with that specific feature. Broker-side auto-creation is
>>> frustrating because it's on by default, and because it can auto-create
>>> topics even if the producer or consumer didn't explicitly ask for that.
>>> Neither of those problems applies to this KIP: producers have to
>>> specifically opt in, and it won't be on by default. Basically, we think
>>> that client-side auto-creation is actually a lot better-- hence this KIP
>>> :)
>>> 
>>> 
>>> 
>>>> 
>>>> 
>>>> But there is
>>>> a functionality which allowed creation of topics if they don't exist in
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> the
>>> 
>>> 
>>> 
>>>> 
>>>> 
>>>> cluster and this behavior could be controlled by a config on the server
>>>> side. Now with this KIP we are allowing producer to make that decision
>>>> without any gateway on the server via configs. Any user who is not aware
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> of
>>> 
>>> 
>>> 
>>>> 
>>>> 
>>>> such changes
>>>> can accidentally create these topics and we are essentially removing a
>>>> config that exists in brokers today to block this accidental creation and
>>>> allowing clients to take control.
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> Again, I hope I'm not misinterpreting, but I don't see how this can be
>>> turned on accidentally. The user would have to specifically turn this on
>>> in the producer by setting the configuration key.
>>> 
>>> 
>>> 
>>>> 
>>>> 
>>>> I still didn't get any positives of not having server side configs? if you
>>>> want to turn them on and allow any client to create topics set the default
>>>> to true
>>>> and allow users who doesn't want to have this feature let them turn them
>>>> off. It will be the exact behavior as it is today , as far as producer is
>>>> concerned. I am not
>>>> understanding why not having server side configs to gateways such a hard
>>>> requirement and this behavior exists today. As far I am concerned this
>>>> breaks the backward compatibility.
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> The downside is that if we wanted to check a server side configuration
>>> before sending the create topics request, the code would be more complex.
>>> The behavior would also not be consistent with how topic auto-creation is
>>> handled in Kafka Streams.
>>> 
>>> 
>>> 
>>> 
>>> In general, it would be nice if we could keep the security and access
>>> control model simple and not introduce a lot of special cases and
>>> exceptions. Kafka has basically converged on using ACLs and
>>> CreateTopicPolicy classes to control who can create topics and where.
>>> Adding more knobs seems like a step backwards, especially when the
>>> proposed knobs don't work consistently across components, and don't
>>> provide true security.
>>> 
>>> 
>>> 
>>> 
>>> Maybe we should support an insecure mode where there are still users and
>>> ACLs. That would help people who don't want to set up Kerberos, etc. but
>>> still want to protect against misconfigured clients or accidents. Hadoop
>>> has something like this, and I think it was useful. NFS also supported
>>> (supports?) a mode where you just pass whatever user ID you want and the
>>> system believes you. These things clearly don't protect against malicious
>>> users, but they can help set up policies when needed.
>>> 
>>> 
>>> 
>>> best,
>>> Colin
>>> 
>>> 
>>>> 
>>>> 
>>>> Thanks,
>>>> Harsha
>>>> 
>>>> 
>>>> 
>>>> On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache. org (
>>>> cmccabe@apache.org ) > wrote:
>>>> 
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> Hi Harsha,
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Thanks for taking a look.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> I'm not sure I follow the discussion about AdminClient.
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> KafkaAdminClient
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> has been around for about 2 years at this point as a public class.
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> There
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> are many programs that use it to automatically create topics. Kafka
>>>>> Streams does this, for example. If any of your users run Kafka
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> Streams,
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> they will be auto-creating topics, regardless of what setting you use
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> for
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> auto.create.topics.enable.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> A big part of the annoyance of auto-topic creation right now is that
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> it is
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> on by default. The new configuration proposed by KIP-487 wouldn't be.
>>>>> Users would have to explicitly opt in to the new behavior of
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> client-side
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> topic creation. If you run without security, you're already putting a
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> huge
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> amount of trust in your users. For example, you trust them not to
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> delete
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> records with the kafka-delete-records. sh ( http://kafka-delete-records.sh/
>>>>> ) command, or delete topics with kafka-topics. sh ( http://kafka-topics.sh/
>>>>> ). Trusting them not to set a certain config value seems minor in
>>>>> comparison, right?
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> best,
>>>>> Colin
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Hi,
>>>>>> Even with policies one needs to implement that, so for every
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> user who
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> doesn't want a producer to create topics or have limits around
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> partitions &
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> replication factor they have to implement a policy. The KIP is changing
>>>>>> the behavior , it might not be introducing
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> the
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> new functionality but it will enable producers to override the create
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> topic
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> config settings on the broker. What I am asking for to provide a
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> config
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> that will disable auto creation of topics and if its enabled set some
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> sane
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> defaults so that clients can create a topic with in those limits. I
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> don't
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> see how this not related to this KIP.
>>>>>> If the server config options as I suggested doesn't interest
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> you at
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> least have a default CreateTopicPolices in place.
>>>>>> To give an example, In our environment we disable the
>>>>>> auto.create.topic.enable and force users to go through a centralized
>>>>>> service as we want capture more details about the topic creation and
>>>>>> requirements. With this KIP, a producer can create a topic with no
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> bounds.
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Another example max.message.size we define that at cluster level
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> and one
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> can override max.messsage.size at topic level, any misconfiguration
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> at
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> this
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> will cause service degradation. Its not always about the rogue
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> clients,
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> users can easily misconfigure and can cause an outage. Again we can talk
>>>>>> about CreateTopicPolicy but without having a
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> default
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> implementation and asking everyone to implement their own while
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> changing
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> the behavior in producer doesn't make sense to me.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Thanks,
>>>>>> Harsha
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me. uk (
>>>>>> ismael@juma.me.uk ) >
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> wrote:
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Harsha,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I mentioned policies and the authorizer. For example, with
>>>>>>> CreateTopicPolicy, you can implement the limits you describe. If
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> you
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> have
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> ideas of how that should be improved, please submit a KIP. My
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> point is
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> that
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> this KIP is not introducing any new functionality with regards to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> what
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> rogue clients can do. It's using the existing protocol that is
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> already
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> exposed via the AdminClient. So, I don't think we need to address
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> it in
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> this KIP. Does that make sense?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Ismael
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> kafka@ harsha. io ( kafka@harsha.io ) >
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Ismael,
>>>>>>> Sure AdminClient can do that and we should've shipped a config or
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> use
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> the
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> existing one to block that. Not all users are yet to upgrade to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> AdminClient
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> and start using that to cause issues yet. In shared environment we
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> should
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> allow server to set sane defaults and not allow every client to go
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> ahead
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> create random no.of topic/partitions and replication factor. Even
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> if
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> the
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> users want to allow topic creation proposed in the KIP , it makes
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> sense to
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> have some guards against the no.of partitions and replication
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> factor.
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Authorizer is not always an answer to block requests and having
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> users
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> set
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> server side configs to protect a multi-tenant environment is
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> required.
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> In a
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> non-secure environment Authorizer is a blunt instrument either you
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> end
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> up
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> blocking everyone or allowing everyone.
>>>>>>> I am asking to have server side that allow clients to create
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> topics or
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> not
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> , if they are allowed set a ceiling on max no.of partitions and
>>>>>>> replication-factor.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -Harsha
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com ( ismaelj@gmail.com )
>>>>>>> > wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Harsha,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Rogue clients can use the admin client to create topics and
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> partitions.
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> ACLs and policies can help in that case as well as this one.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Ismael
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha. io (
>>>>>>> kafka@harsha.io ) >
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Justine,
>>>>>>> Thanks for the KIP.
>>>>>>> "When server-side auto-creation is disabled, client-side
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> auto-creation
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> will try to use client-side configurations"
>>>>>>> If I understand correctly, this KIP is removing any server-side
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> blocking
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> client auto creation of topic?
>>>>>>> if so this will present potential issue of rogue client creating
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> ton of
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> topic-partitions and potentially bringing down the service for
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> everyone
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> or
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> degrade the service itself.
>>>>>>> By reading the KIP its not clear to me that there is a clear way to
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> block
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> auto creation topics of all together from clients by server side
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> config.
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Server side configs of default topic, partitions should take higher
>>>>>>> precedence and client shouldn't be able to create a topic with
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> higher
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> no.of
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> partitions, replication than what server config specifies.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Harsha
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi all,
>>>>>>> I made some changes to the KIP. Hopefully this configuration change
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> will
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> make things a little clearer.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/ ) KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Please let me know if you have any feedback or questions!
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thank you,
>>>>>>> Justine
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache. org (
>>>>>>> cmccabe@apache.org ) >
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Mickael,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I think you bring up a good point. It would be better if we didn't
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> ever
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> have to set up client-side configuration for this feature, and
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> KIP-464
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> would let us skip this entirely.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Mickael,
>>>>>>> I agree that KIP-464 works on newer brokers, but I was a bit
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> worried
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> how
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> things would play out on older brokers that* do not *have KIP 464
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> included.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Is it enough to throw an error in this case when producer configs
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> are
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> not
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> specified?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I think the right thing to do would be to log an error message in
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> the
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> client. We will need to have that capability in any case, to cover
>>>>>>> scenarios like the client trying to auto-create a topic that they
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> don't
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> have permission to create. Or a client trying to create a topic on
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> a
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> broker
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> so old that CreateTopicsRequest is not supported.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> The big downside to relying on KIP-464 is that it is a very recent
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> feature
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- so recent that it hasn't even made its way to any official
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> Apache
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> release. It's scheduled for the upcoming 2.4 release in a few
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> months.
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> So if you view this KIP as a step towards removing broker-side
>>>>>>> auto-create, you might want to support older brokers just to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> accelerate
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> adoption, and hasten the day when we can finally flip broker-side
>>>>>>> auto-create to off (or even remove it entirely).
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I have to agree, though, that having client-side configurations for
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> number
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> of partitions and replication factor is messy. Maybe it would be
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> worth
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> it
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> to restrict support to post-KIP-464 brokers, if we could avoid
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> creating
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> more configs.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> best,
>>>>>>> Colin
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> mickael. maison@ gmail. com ( mickael.maison@gmail.com )
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Justine,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> We can rely on KIP-464 which allows to omit the partition count or
>>>>>>> replication factor when creating a topic. In that case, the broker
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> defaults
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> are used.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Michael,
>>>>>>> That makes sense to me!
>>>>>>> To clarify, in the current state of the KIP, the producer does not
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> rely
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> on
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the broker to autocreate--if the broker's config is disabled, then
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> producer can autocreate on its own with a create topic request (the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> same
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> type of request the admin client uses).
>>>>>>> However, if both configs are enabled, the broker will autocreate
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> through
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> a
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> metadata request before the producer gets a chance. Of course, the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> way
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> avoid this, is to do as you suggested, and set
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> "allow_auto_topic_creation" field to false.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I think the only thing we need to be careful with in this setup is
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> without
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> KIP 464, we can not use broker defaults for this topic. A user
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> needs
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> specify the number of partition and replication factor in the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> config.
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> An
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> alternative to this is to have coded defaults for when these
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> configs
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> are
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> unspecified, but it is not immediately apparent what these defaults
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> should
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> be.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks again for reading my KIP,
>>>>>>> Justine
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> mickael. maison@ gmail. com ( mickael.maison@gmail.com )
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Justine,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks for the response!
>>>>>>> In my opinion, it would be better if the producer did not rely at
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> all
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> on the broker auto create feature as this is what we're aiming to
>>>>>>> deprecate. When requesting metadata we can set the
>>>>>>> "allow_auto_topic_creation" field to false to avoid the broker auto
>>>>>>> creation. Then if the topic is not existing, send a
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> CreateTopicRequest.
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> What do you think?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Currently the way it is implemented, the broker auto-creation
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> configuration
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> takes precedence. The producer will not use the CreateTopics
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> request.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> (Technically it can--but the topic will already be created
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> through
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> broker, so it will never try to create the topic.) It is possible
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> to
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> change this however, and I'd be happy to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> discuss
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> benefits of this alternative.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thank you,
>>>>>>> Justine
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> mickael. maison@ gmail. com ( mickael.maison@gmail.com ) >
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Justine,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks for the KIP!
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> In case auto creation is enabled on both the client and server,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> will
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the producer still use the AdminClient (CreateTopics request)
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> create topics? and not rely on the broker auto create. I'm
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> guessing the
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> answer is yes but can you make it explicit.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thank you
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi,
>>>>>>> Just a friendly reminder to take a look at this KIP if you
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> have
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> time.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I was thinking about broker vs. client default precedence,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> and I
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> think it
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> makes sense to keep the broker as the default used when both
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> client-side
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> and broker-side defaults are configured. The idea is that
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> there
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> would be
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> pretty clear documentation, and that many systems with
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> configurations
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> that
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the client could not change would likely have the auto-create
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> default
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> off.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> (In cloud for example).
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> It also seems like in most cases, the consumer config
>>>>>>> ' allow. auto. create. topics ( http://allow.auto.create.topics/ ) ' was
>>>>>>> created to actually prevent
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> creation
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> of
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> topics, so the loss of creation functionality will not be a
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> big
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> problem.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I'm happy to discuss any other compatibility problems or
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> components
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> of
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> this KIP.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thank you,
>>>>>>> Justine
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> jolshan@ confluent. io ( jolshan@confluent.io )
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hello all,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I was looking at this KIP again, and there is a decision I
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> made
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> that I
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> think is worth discussing.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> In the case where both the broker and producer's
>>>>>>> 'auto.create.topics.enable' are set to true, we have to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> choose
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> either
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> broker configs or the producer configs for the replication
>>>>>>> factor/partitions.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Currently, the decision is to have the broker defaults take
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> precedence.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> (It is easier to do this in the implementation.) It also
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> makes
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> some
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> sense
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> for this behavior to take precedence since this behavior
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> already
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> occurs as
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the default.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> However, I was wondering if it would be odd for those who
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> can
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> only
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> see
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> producer side to set configs for replication
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> factor/partitions
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> and
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> see
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> different results. Currently the documentation for the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> config
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> states
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> that
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the config values are only used when the broker config is
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> not
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> enabled,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> but
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> this might not always be clear to the user. Changing the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> code
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> have
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> producer's configurations take precedence is possible, but
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> was
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wondering
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> what everyone thought.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thank you,
>>>>>>> Justine
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Just a quick update--
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> It seems that enabling both the broker and producer
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> configs
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> works
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> fine,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> except that the broker configurations for partitions,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> replication
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> factor
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> take precedence.
>>>>>>> I don't know if that is something we would want to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> change, but
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I'll be
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> updating the KIP for now to reflect this. Perhaps we would
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> want to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> add more
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> to the documentation of the the producer configs to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> clarify.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thank you,
>>>>>>> Justine
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Colin,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks for looking at the KIP. I can definitely add to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> title
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> make
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> it more clear.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> It makes sense that both configurations could be turned
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> on
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> since
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> there
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> are many cases where the user can not control the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> server-side
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> configurations. I was a little concerned about how both
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> interacting
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> would
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> work out -- if there would be to many requests for new
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> topics,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> for
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> example.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> But it since it does make sense to allow both
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> configurations
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> enabled, I
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> will test out some scenarios and I'll change the KIP to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> support
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> this.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I also agree with documentation about distinguishing the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> differences. I
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> was having some trouble with the wording but I like the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> phrases
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> "server-side" and "client-side." That's a good
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> distinction I
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> can
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> use
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> when
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> describing.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I'll try to update the KIP soon keeping everyone's input
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> in
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> mind.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Justine
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> cmccabe@ apache. org ( cmccabe@apache.org )
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Justine,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks for the KIP. This seems like a good step towards
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> removing
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> server-side topic auto-creation.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> We should add included "client-side" to the title of
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the KIP
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> somewhere,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> to make it clear that we're talking about client-side
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> auto
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> creation.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> The KIP says:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> In order to automatically create topics with the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> producer, the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> producer's
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> auto.create.topics.enable config must be set to true
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> and
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> broker
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> config should be set to false
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> From a user's point of view, this seems
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> counter-intuitive.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> In
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> order to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> auto-create topics the broker's
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> auto.create.topics.enable
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> config
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> should be
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> set to false? It seems like the server-side
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> auto-create is
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> unrelated to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the client-side auto-create. We could have both turned
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> on
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> (and
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I'm
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> sure
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> that in the real world, people will try this
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> configuration...)
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> There's no
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> reason not to support this, I think.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> We should add some documentation explaining the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> difference
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> between
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> server-side and client-side auto-creation. Without
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> documentation,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> an admin
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> might think that they had disabled all forms of
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> auto-creation by
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> setting
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the -side setting to false-- but this is not the case,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> of
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> course.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> best,
>>>>>>> Colin
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Dhruvil,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks for reading the KIP!
>>>>>>> That was the general idea for deprecation. We would
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> log a
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> warning
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> when the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> config is enabled on the broker.
>>>>>>> I also believe that there would be a change to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> documentation.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> If there is anything else that should be done, please
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> let
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> me
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> know!
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Justine
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> dhruvil@ confluent. io ( dhruvil@confluent.io ) >
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Justine,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks for the KIP, this is great!
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Could you add some more information about what
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> deprecating
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> broker
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> configuration means? Would we log a warning in the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> logs
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> when
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> auto
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> topic
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> creation is enabled on the broker, for example?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Dhruvil
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan <
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> jolshan@ confluent. io ( jolshan@confluent.io ) >
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hello all,
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I'd like to start a discussion thread for KIP-487. This KIP plans
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> to
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> deprecate the current system of
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> auto-creating
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> topics
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> through requests to the metadata and give the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> producer the
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> ability to
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> automatically create topics instead.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> More information can be found here:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/ ) KIP-487%3A+Automatic+Topic+Creation+on+Producer
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thank you,
>>>>>>> Justine Olshan
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Colin McCabe <cm...@apache.org>.

On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> Hi Colin,
> "Hmm... I'm not sure I follow.  Users don't have to build their own
> tooling, right?  They can use any of the shell scripts that we've shipped
> in the last few releases.  For example, if any of your users run it, this
> shell script will delete all of the topics from your non-security-enabled
> cluster:
> 
> ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list 2>/dev/null
> | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete
> --topic
> 
> They will need to fill in the correct bootstrap servers list, of course,
> not localhost.  This deletion script will work on some pretty old brokers,
> even back to the 0.10 releases.  It seems a little odd to trust your users
> with this power, but not trust them to avoid changing a particular
> configuration key."
> 
> The above will blocked by the server if we set delete.topic.enable to false
> and thats exactly what I am asking for.

Hi Harsha,

I was wondering if someone was going to bring up that configuration :)

it's an interesting complication, but globally disabling topic deletion is not very practical for most use-cases. 

In any case, there are plenty of other bad things that users with full permissions can do that aren't blocked by any server configuration.  For example, they can delete every record in every topic.  I can write a script for that too, and there's no server configuration you can set to disable it.  Or I could simply create hundreds of thousands of topics, until cluster performance becomes unacceptable (this will be even more of a problem if someone configured delete.topic.enable as false).  Or publish bad data to every topic, etc. etc.

The point I'm trying to make here is that you can't rely on these kind of server-side configurations for security.  At most, they're a way to set up certain very simple policies.  But the policies are so simple that they're hardly ever useful any more.

For example, if the problem you want to solve is that you want a user to only be able to create 50 topics and not delete anyone else's topics, you can solve that with a CreateTopicsPolicy that limits the number of topics, and some ACLs.  There's no combination of auto.create.topics.enable and delete.topic.enable that will help here. 

> 
> "The downside is that if we wanted to check a server side configuration
> before sending the create topics request, the code would be more complex.
> The behavior would also not be consistent with how topic auto-creation is
> handled in Kafka Streams."
> 
> I am not sure why you need to check server side configuration before
> sending create topics request. A user enables producer side config to
> create topics.
> Producer sends a request to the broker and if the broker has
> auto.topic.create.enable to true (default) it will allow creation of
> topics. If it set to false it returns error back to the client.

auto.topic.create.enable has never affected CreateTopicsRequest.  If you submit a CreateTopicsRequest and you are authorized, a topic will be created, regardless of what the value of auto.topic.create.enable is.  This behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?)

> I don't see how this behavior will be different in Kafka streams. By
> default server allows the topic creation and with this KIP, It will only
> allow creation of topic when both producer and server side are turned on.
> Its exactly the same behavior in KIP-361.

I suggest running a streams job as a test.  It creates the topics it needs using AdminClient.  The value of auto.topic.create.enable is not relevant.  Whether it is true or false, the topics will still be created.

> 
> "In general, it would be nice if we could keep the security and access
> control model simple and not introduce a lot of special cases and
> exceptions.  Kafka has basically converged on using ACLs and
> CreateTopicPolicy classes to control who can create topics and where.
> Adding more knobs seems like a step backwards, especially when the 
> proposed knobs don't work consistently across components, and don't provide true
> security." This is not about access control at all. Shipping sane defaults should 
> be prioritized.

I don't think the default is really the question here.  I think everyone agrees that the default for client-side auto-topic creation should be off.

The scenarios that you're describing (such as dealing with a poorly configured client that tries to create topics it should not) do seem to be about access control.  

> We keep talking about CreateTopicPolicy and yet we don't have default 
> one and asking every user of Kafka implement their own doesn't make sense 
> here.

The point of CreateTopicPolicy is exactly that you can implement your own, though.  It's a way of customizing what the policy is in your specific cluster.

I agree that most people don't want to write a custom policy.  But most of those people are satisfied with just setting up ACLs to set up simple policies like this user can create topics, that user can't, etc.  That's why I suggested adding support for ACLs to insecure clusters might be helpful.

Also, just as a side note, wouldn't it would be impossible for us to specify a default CreateTopicsPolicy that did anything at this point anyway without breaking backwards compatibility?  Maybe I'm misunderstanding the proposal.

> I am asking about exact behavior that we shipped for consumer side
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
> 
> I still didn't get any response why this behavior shouldn't be exactly like
> Kafka consumer and why do we want to have producer to overrider server side
> config and while not allowing consumer to do so.
> We are not even allowing the same contract and this will cause more
> confusion from the users standpoint.

Hmm.  The consumer already has a way to override the server side configuration.  See KIP-361: Add Consumer Configuration to Disable Auto Topic Creation.  This lets the consumer skip auto-creating topics, even if auto-creation is enabled on the broker.

To be fair, the KIP should probably discuss why we don't want to implement client-side topic creation in the consumer in "rejected alternatives."  Maybe Justine can add more context here in the KIP.

The last time we talked about this, the reasoning was that we wanted to eventually get rid of consumer-side auto-topic creation entirely, and so it wasn't worth implementing an improved way of doing it.  Producer side auto topic-creation, in contrast, will probably stick around, although we'd like to deprecate and remove the problematic server-side mechanism over time.

best,
Colin

> 
> 
> On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe <cm...@apache.org> wrote:
> 
> > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > > Not sure how the AdminClient applies here, It is an external API and
> > > is not part of KafkaProducer so any user who updates to latest version of
> > > Kafka with this feature get to create the topics.
> > > They have to build a tooling around AdminClient allow themselves to
> > create
> > > topics.
> >
> > Hi Harsha,
> >
> > Hmm... I'm not sure I follow.  Users don't have to build their own
> > tooling, right?  They can use any of the shell scripts that we've shipped
> > in the last few releases.  For example, if any of your users run it, this
> > shell script will delete all of the topics from your non-security-enabled
> > cluster:
> >
> > ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list 2>/dev/null
> > | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete
> > --topic
> >
> > They will need to fill in the correct bootstrap servers list, of course,
> > not localhost.  This deletion script will work on some pretty old brokers,
> > even back to the 0.10 releases.  It seems a little odd to trust your users
> > with this power, but not trust them to avoid changing a particular
> > configuration key.
> >
> > > There is no behavior in Kafka producer that allowed users to
> > > delete the topics or delete the records. So citing them as an
> > > example doesn't makes sense in this context.
> >
> > I think Kafka Streams is relevant here.  After all, it's software that we
> > ship as part of the official Kafka release.  And it auto-creates topics
> > even when auto.create.topics.enable is set to false on the broker.
> >
> > I think that auto.create.topics.enable was never intended as a security
> > setting to control access.  It was intended as a way to disable the
> > broker-side auto-create feature specifically, because there were some
> > problems with that specific feature.  Broker-side auto-creation is
> > frustrating because it's on by default, and because it can auto-create
> > topics even if the producer or consumer didn't explicitly ask for that.
> > Neither of those problems applies to this KIP: producers have to
> > specifically opt in, and it won't be on by default.  Basically, we think
> > that client-side auto-creation is actually a lot better-- hence this KIP :)
> >
> > > But there is
> > > a functionality which allowed creation of topics if they don't exist in
> > the
> > > cluster and this behavior could be controlled by a config on the server
> > > side. Now with this KIP we are allowing producer to make that decision
> > > without any gateway on the server via configs. Any user who is not aware
> > of
> > > such changes
> > > can accidentally create these topics and we are essentially removing a
> > > config that exists in brokers today to block this accidental creation and
> > > allowing clients to take control.
> >
> > Again, I hope I'm not misinterpreting, but I don't see how this can be
> > turned on accidentally.  The user would have to specifically turn this on
> > in the producer by setting the configuration key.
> >
> > >       I still didn't get any positives of not having server side configs?
> > > if you want to turn them on and allow any client to create topics set the
> > > default to true
> > > and allow users who doesn't want to have this feature let them turn them
> > > off. It will be the exact behavior as it is today, as far as producer is
> > > concerned. I am not
> > > understanding why not having server side configs to gateways such a hard
> > > requirement and this behavior exists today. As far I am concerned this
> > > breaks the backward compatibility.
> >
> > The downside is that if we wanted to check a server side configuration
> > before sending the create topics request, the code would be more complex.
> > The behavior would also not be consistent with how topic auto-creation is
> > handled in Kafka Streams.
> >
> > In general, it would be nice if we could keep the security and access
> > control model simple and not introduce a lot of special cases and
> > exceptions.  Kafka has basically converged on using ACLs and
> > CreateTopicPolicy classes to control who can create topics and where.
> > Adding more knobs seems like a step backwards, especially when the proposed
> > knobs don't work consistently across components, and don't provide true
> > security.
> >
> > Maybe we should support an insecure mode where there are still users and
> > ACLs.  That would help people who don't want to set up Kerberos, etc. but
> > still want to protect against misconfigured clients or accidents.  Hadoop
> > has something like this, and I think it was useful.  NFS also supported
> > (supports?) a mode where you just pass whatever user ID you want and the
> > system believes you.  These things clearly don't protect against malicious
> > users, but they can help set up policies when needed.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > Hi Harsha,
> > > >
> > > > Thanks for taking a look.
> > > >
> > > > I'm not sure I follow the discussion about AdminClient.
> > KafkaAdminClient
> > > > has been around for about 2 years at this point as a public class.
> > There
> > > > are many programs that use it to automatically create topics.  Kafka
> > > > Streams does this, for example.  If any of your users run Kafka
> > Streams,
> > > > they will be auto-creating topics, regardless of what setting you use
> > for
> > > > auto.create.topics.enable.
> > > >
> > > > A big part of the annoyance of auto-topic creation right now is that
> > it is
> > > > on by default.  The new configuration proposed by KIP-487 wouldn't be.
> > > > Users would have to explicitly opt in to the new behavior of
> > client-side
> > > > topic creation.  If you run without security, you're already putting a
> > huge
> > > > amount of trust in your users.  For example, you trust them not to
> > delete
> > > > records with the kafka-delete-records.sh command, or delete topics with
> > > > kafka-topics.sh.  Trusting them not to set a certain config value seems
> > > > minor in comparison, right?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > > > > Hi,
> > > > >     Even with policies one needs to implement that, so for every
> > user who
> > > > > doesn't want a producer to create topics or have limits around
> > > > partitions &
> > > > > replication factor they have to implement a policy.
> > > > >       The KIP is changing the behavior , it might not be introducing
> > the
> > > > > new functionality but it will enable producers to override the create
> > > > topic
> > > > > config settings on the broker. What I am asking for to provide a
> > config
> > > > > that will disable auto creation of topics and if its enabled set some
> > > > sane
> > > > > defaults so that clients can create a topic with in those limits. I
> > don't
> > > > > see how this not related to this KIP.
> > > > >      If the server config options as I suggested doesn't interest
> > you at
> > > > > least have a default CreateTopicPolices in place.
> > > > >        To give an example, In our environment we disable the
> > > > > auto.create.topic.enable and force users to go through a centralized
> > > > > service as we want capture more details about the topic creation and
> > > > > requirements. With this KIP, a producer can create a topic with no
> > > > bounds.
> > > > >  Another example max.message.size we define that at cluster level
> > and one
> > > > > can override max.messsage.size at topic level, any misconfiguration
> > at
> > > > this
> > > > > will cause service degradation.  Its not always about the rogue
> > clients,
> > > > > users can easily misconfigure and can cause an outage.
> > > > > Again we can talk about CreateTopicPolicy but without having a
> > default
> > > > > implementation and asking everyone to implement their own while
> > changing
> > > > > the behavior in producer  doesn't make sense to me.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > >
> > > > > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > > >
> > > > > > Hi Harsha,
> > > > > >
> > > > > > I mentioned policies and the authorizer. For example, with
> > > > > > CreateTopicPolicy, you can implement the limits you describe. If
> > you
> > > > have
> > > > > > ideas of how that should be improved, please submit a KIP. My
> > point is
> > > > that
> > > > > > this KIP is not introducing any new functionality with regards to
> > what
> > > > > > rogue clients can do. It's using the existing protocol that is
> > already
> > > > > > exposed via the AdminClient. So, I don't think we need to address
> > it in
> > > > > > this KIP. Does that make sense?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <
> > kafka@harsha.io>
> > > > > > wrote:
> > > > > >
> > > > > > Ismael,
> > > > > > Sure AdminClient can do that and we should've shipped a config or
> > use
> > > > the
> > > > > > existing one to block that. Not all users are yet to upgrade to
> > > > AdminClient
> > > > > > and start using that to cause issues yet. In shared environment we
> > > > should
> > > > > > allow server to set sane defaults and not allow every client to go
> > > > ahead
> > > > > > create random no.of topic/partitions and replication factor. Even
> > if
> > > > the
> > > > > > users want to allow topic creation proposed in the KIP , it makes
> > > > sense to
> > > > > > have some guards against the no.of partitions and replication
> > factor.
> > > > > > Authorizer is not always an answer to block requests and having
> > users
> > > > set
> > > > > > server side configs to protect a multi-tenant environment is
> > required.
> > > > In a
> > > > > > non-secure environment Authorizer is a blunt instrument either you
> > end
> > > > up
> > > > > > blocking everyone or allowing everyone.
> > > > > > I am asking to have server side that allow clients to create
> > topics or
> > > > not
> > > > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > > > replication-factor.
> > > > > >
> > > > > > -Harsha
> > > > > >
> > > > > > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> > > > > >
> > > > > > Harsha,
> > > > > >
> > > > > > Rogue clients can use the admin client to create topics and
> > partitions.
> > > > > > ACLs and policies can help in that case as well as this one.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi Justine,
> > > > > > Thanks for the KIP.
> > > > > > "When server-side auto-creation is disabled, client-side
> > auto-creation
> > > > > > will try to use client-side configurations"
> > > > > > If I understand correctly, this KIP is removing any server-side
> > > > blocking
> > > > > > client auto creation of topic?
> > > > > > if so this will present potential issue of rogue client creating
> > ton of
> > > > > > topic-partitions and potentially bringing down the service for
> > everyone
> > > > > >
> > > > > > or
> > > > > >
> > > > > > degrade the service itself.
> > > > > > By reading the KIP its not clear to me that there is a clear way to
> > > > block
> > > > > > auto creation topics of all together from clients by server side
> > > > config.
> > > > > > Server side configs of default topic, partitions should take higher
> > > > > > precedence and client shouldn't be able to create a topic with
> > higher
> > > > > >
> > > > > > no.of
> > > > > >
> > > > > > partitions, replication than what server config specifies.
> > > > > >
> > > > > > Thanks,
> > > > > > Harsha
> > > > > >
> > > > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <
> > jolshan@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > > I made some changes to the KIP. Hopefully this configuration change
> > > > > >
> > > > > > will
> > > > > >
> > > > > > make things a little clearer.
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > > > >
> > > > > > Please let me know if you have any feedback or questions!
> > > > > >
> > > > > > Thank you,
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I think you bring up a good point. It would be better if we didn't
> > ever
> > > > > > have to set up client-side configuration for this feature, and
> > KIP-464
> > > > > > would let us skip this entirely.
> > > > > >
> > > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > > I agree that KIP-464 works on newer brokers, but I was a bit
> > worried
> > > > > >
> > > > > > how
> > > > > >
> > > > > > things would play out on older brokers that* do not *have KIP 464
> > > > > >
> > > > > > included.
> > > > > >
> > > > > > Is it enough to throw an error in this case when producer configs
> > are
> > > > > >
> > > > > > not
> > > > > >
> > > > > > specified?
> > > > > >
> > > > > > I think the right thing to do would be to log an error message in
> > the
> > > > > > client. We will need to have that capability in any case, to cover
> > > > > > scenarios like the client trying to auto-create a topic that they
> > don't
> > > > > > have permission to create. Or a client trying to create a topic on
> > a
> > > > > >
> > > > > > broker
> > > > > >
> > > > > > so old that CreateTopicsRequest is not supported.
> > > > > >
> > > > > > The big downside to relying on KIP-464 is that it is a very recent
> > > > > >
> > > > > > feature
> > > > > >
> > > > > > -- so recent that it hasn't even made its way to any official
> > Apache
> > > > > > release. It's scheduled for the upcoming 2.4 release in a few
> > months.
> > > > > >
> > > > > > So if you view this KIP as a step towards removing broker-side
> > > > > > auto-create, you might want to support older brokers just to
> > accelerate
> > > > > > adoption, and hasten the day when we can finally flip broker-side
> > > > > > auto-create to off (or even remove it entirely).
> > > > > >
> > > > > > I have to agree, though, that having client-side configurations for
> > > > > >
> > > > > > number
> > > > > >
> > > > > > of partitions and replication factor is messy. Maybe it would be
> > worth
> > > > > >
> > > > > > it
> > > > > >
> > > > > > to restrict support to post-KIP-464 brokers, if we could avoid
> > creating
> > > > > > more configs.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> > > > > >
> > > > > > mickael.maison@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > We can rely on KIP-464 which allows to omit the partition count or
> > > > > > replication factor when creating a topic. In that case, the broker
> > > > > >
> > > > > > defaults
> > > > > >
> > > > > > are used.
> > > > > >
> > > > > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <
> > jolshan@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > Michael,
> > > > > > That makes sense to me!
> > > > > > To clarify, in the current state of the KIP, the producer does not
> > > > > >
> > > > > > rely
> > > > > >
> > > > > > on
> > > > > >
> > > > > > the broker to autocreate--if the broker's config is disabled, then
> > > > > >
> > > > > > the
> > > > > >
> > > > > > producer can autocreate on its own with a create topic request (the
> > > > > >
> > > > > > same
> > > > > >
> > > > > > type of request the admin client uses).
> > > > > > However, if both configs are enabled, the broker will autocreate
> > > > > >
> > > > > > through
> > > > > >
> > > > > > a
> > > > > >
> > > > > > metadata request before the producer gets a chance. Of course, the
> > way
> > > > > >
> > > > > > to
> > > > > >
> > > > > > avoid this, is to do as you suggested, and set
> > > > > >
> > > > > > the
> > > > > >
> > > > > > "allow_auto_topic_creation" field to false.
> > > > > >
> > > > > > I think the only thing we need to be careful with in this setup is
> > > > > >
> > > > > > without
> > > > > >
> > > > > > KIP 464, we can not use broker defaults for this topic. A user
> > needs
> > > > > >
> > > > > > to
> > > > > >
> > > > > > specify the number of partition and replication factor in the
> > config.
> > > > > >
> > > > > > An
> > > > > >
> > > > > > alternative to this is to have coded defaults for when these
> > > > > >
> > > > > > configs
> > > > > >
> > > > > > are
> > > > > >
> > > > > > unspecified, but it is not immediately apparent what these defaults
> > > > > >
> > > > > > should
> > > > > >
> > > > > > be.
> > > > > >
> > > > > > Thanks again for reading my KIP,
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > > > > >
> > > > > > mickael.maison@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > Thanks for the response!
> > > > > > In my opinion, it would be better if the producer did not rely at
> > > > > >
> > > > > > all
> > > > > >
> > > > > > on the broker auto create feature as this is what we're aiming to
> > > > > > deprecate. When requesting metadata we can set the
> > > > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > > > creation. Then if the topic is not existing, send a
> > CreateTopicRequest.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > > > > >
> > > > > > jolshan@confluent.io>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Currently the way it is implemented, the broker auto-creation
> > > > > >
> > > > > > configuration
> > > > > >
> > > > > > takes precedence. The producer will not use the CreateTopics
> > > > > >
> > > > > > request.
> > > > > >
> > > > > > (Technically it can--but the topic will already be created
> > > > > >
> > > > > > through
> > > > > >
> > > > > > the
> > > > > >
> > > > > > broker, so it will never try to create the topic.) It is possible
> > to
> > > > > > change this however, and I'd be happy to
> > > > > >
> > > > > > discuss
> > > > > >
> > > > > > the
> > > > > >
> > > > > > benefits of this alternative.
> > > > > >
> > > > > > Thank you,
> > > > > > Justine
> > > > > >
> > > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > > > > >
> > > > > > mickael.maison@gmail.com>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > Thanks for the KIP!
> > > > > >
> > > > > > In case auto creation is enabled on both the client and server,
> > > > > >
> > > > > > will
> > > > > >
> > > > > > the producer still use the AdminClient (CreateTopics request)
> > > > > >
> > > > > > to
> > > > > >
> > > > > > create topics? and not rely on the broker auto create. I'm
> > guessing the
> > > > > > answer is yes but can you make it explicit.
> > > > > >
> > > > > > Thank you
> > > > > >
> > > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > > > > >
> > > > > > jolshan@confluent.io>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > > Just a friendly reminder to take a look at this KIP if you
> > > > > >
> > > > > > have
> > > > > >
> > > > > > the
> > > > > >
> > > > > > time.
> > > > > >
> > > > > > I was thinking about broker vs. client default precedence,
> > > > > >
> > > > > > and I
> > > > > >
> > > > > > think it
> > > > > >
> > > > > > makes sense to keep the broker as the default used when both
> > > > > >
> > > > > > client-side
> > > > > >
> > > > > > and broker-side defaults are configured. The idea is that
> > > > > >
> > > > > > there
> > > > > >
> > > > > > would be
> > > > > >
> > > > > > pretty clear documentation, and that many systems with
> > > > > >
> > > > > > configurations
> > > > > >
> > > > > > that
> > > > > >
> > > > > > the client could not change would likely have the auto-create
> > > > > >
> > > > > > default
> > > > > >
> > > > > > off.
> > > > > >
> > > > > > (In cloud for example).
> > > > > >
> > > > > > It also seems like in most cases, the consumer config
> > > > > > 'allow.auto.create.topics' was created to actually prevent
> > > > > >
> > > > > > the
> > > > > >
> > > > > > creation
> > > > > >
> > > > > > of
> > > > > >
> > > > > > topics, so the loss of creation functionality will not be a
> > > > > >
> > > > > > big
> > > > > >
> > > > > > problem.
> > > > > >
> > > > > > I'm happy to discuss any other compatibility problems or
> > > > > >
> > > > > > components
> > > > > >
> > > > > > of
> > > > > >
> > > > > > this KIP.
> > > > > >
> > > > > > Thank you,
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > > > > >
> > > > > > jolshan@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > I was looking at this KIP again, and there is a decision I
> > > > > >
> > > > > > made
> > > > > >
> > > > > > that I
> > > > > >
> > > > > > think is worth discussing.
> > > > > >
> > > > > > In the case where both the broker and producer's
> > > > > > 'auto.create.topics.enable' are set to true, we have to
> > > > > >
> > > > > > choose
> > > > > >
> > > > > > either
> > > > > >
> > > > > > the
> > > > > >
> > > > > > broker configs or the producer configs for the replication
> > > > > > factor/partitions.
> > > > > >
> > > > > > Currently, the decision is to have the broker defaults take
> > > > > >
> > > > > > precedence.
> > > > > >
> > > > > > (It is easier to do this in the implementation.) It also
> > > > > >
> > > > > > makes
> > > > > >
> > > > > > some
> > > > > >
> > > > > > sense
> > > > > >
> > > > > > for this behavior to take precedence since this behavior
> > > > > >
> > > > > > already
> > > > > >
> > > > > > occurs as
> > > > > >
> > > > > > the default.
> > > > > >
> > > > > > However, I was wondering if it would be odd for those who
> > > > > >
> > > > > > can
> > > > > >
> > > > > > only
> > > > > >
> > > > > > see
> > > > > >
> > > > > > the
> > > > > >
> > > > > > producer side to set configs for replication
> > > > > >
> > > > > > factor/partitions
> > > > > >
> > > > > > and
> > > > > >
> > > > > > see
> > > > > >
> > > > > > different results. Currently the documentation for the
> > > > > >
> > > > > > config
> > > > > >
> > > > > > states
> > > > > >
> > > > > > that
> > > > > >
> > > > > > the config values are only used when the broker config is
> > > > > >
> > > > > > not
> > > > > >
> > > > > > enabled,
> > > > > >
> > > > > > but
> > > > > >
> > > > > > this might not always be clear to the user. Changing the
> > > > > >
> > > > > > code
> > > > > >
> > > > > > to
> > > > > >
> > > > > > have
> > > > > >
> > > > > > the
> > > > > >
> > > > > > producer's configurations take precedence is possible, but
> > > > > >
> > > > > > I
> > > > > >
> > > > > > was
> > > > > >
> > > > > > wondering
> > > > > >
> > > > > > what everyone thought.
> > > > > >
> > > > > > Thank you,
> > > > > > Justine
> > > > > >
> > > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > > > > >
> > > > > > jolshan@confluent.io>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Just a quick update--
> > > > > >
> > > > > > It seems that enabling both the broker and producer
> > > > > >
> > > > > > configs
> > > > > >
> > > > > > works
> > > > > >
> > > > > > fine,
> > > > > >
> > > > > > except that the broker configurations for partitions,
> > > > > >
> > > > > > replication
> > > > > >
> > > > > > factor
> > > > > >
> > > > > > take precedence.
> > > > > > I don't know if that is something we would want to
> > > > > >
> > > > > > change, but
> > > > > >
> > > > > > I'll be
> > > > > >
> > > > > > updating the KIP for now to reflect this. Perhaps we would
> > > > > >
> > > > > > want to
> > > > > >
> > > > > > add more
> > > > > >
> > > > > > to the documentation of the the producer configs to
> > > > > >
> > > > > > clarify.
> > > > > >
> > > > > > Thank you,
> > > > > > Justine
> > > > > >
> > > > > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > > > > >
> > > > > > jolshan@confluent.io>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi Colin,
> > > > > >
> > > > > > Thanks for looking at the KIP. I can definitely add to
> > > > > >
> > > > > > the
> > > > > >
> > > > > > title
> > > > > >
> > > > > > to
> > > > > >
> > > > > > make
> > > > > >
> > > > > > it more clear.
> > > > > >
> > > > > > It makes sense that both configurations could be turned
> > > > > >
> > > > > > on
> > > > > >
> > > > > > since
> > > > > >
> > > > > > there
> > > > > >
> > > > > > are many cases where the user can not control the
> > > > > >
> > > > > > server-side
> > > > > >
> > > > > > configurations. I was a little concerned about how both
> > > > > >
> > > > > > interacting
> > > > > >
> > > > > > would
> > > > > >
> > > > > > work out -- if there would be to many requests for new
> > > > > >
> > > > > > topics,
> > > > > >
> > > > > > for
> > > > > >
> > > > > > example.
> > > > > >
> > > > > > But it since it does make sense to allow both
> > > > > >
> > > > > > configurations
> > > > > >
> > > > > > enabled, I
> > > > > >
> > > > > > will test out some scenarios and I'll change the KIP to
> > > > > >
> > > > > > support
> > > > > >
> > > > > > this.
> > > > > >
> > > > > > I also agree with documentation about distinguishing the
> > > > > >
> > > > > > differences. I
> > > > > >
> > > > > > was having some trouble with the wording but I like the
> > > > > >
> > > > > > phrases
> > > > > >
> > > > > > "server-side" and "client-side." That's a good
> > > > > >
> > > > > > distinction I
> > > > > >
> > > > > > can
> > > > > >
> > > > > > use
> > > > > >
> > > > > > when
> > > > > >
> > > > > > describing.
> > > > > >
> > > > > > I'll try to update the KIP soon keeping everyone's input
> > > > > >
> > > > > > in
> > > > > >
> > > > > > mind.
> > > > > >
> > > > > > Thanks,
> > > > > > Justine
> > > > > >
> > > > > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > > > > >
> > > > > > cmccabe@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > Thanks for the KIP. This seems like a good step towards
> > > > > >
> > > > > > removing
> > > > > >
> > > > > > server-side topic auto-creation.
> > > > > >
> > > > > > We should add included "client-side" to the title of
> > > > > >
> > > > > > the KIP
> > > > > >
> > > > > > somewhere,
> > > > > >
> > > > > > to make it clear that we're talking about client-side
> > > > > >
> > > > > > auto
> > > > > >
> > > > > > creation.
> > > > > >
> > > > > > The KIP says:
> > > > > >
> > > > > > In order to automatically create topics with the
> > > > > >
> > > > > > producer, the
> > > > > >
> > > > > > producer's
> > > > > >
> > > > > > auto.create.topics.enable config must be set to true
> > > > > >
> > > > > > and
> > > > > >
> > > > > > the
> > > > > >
> > > > > > broker
> > > > > >
> > > > > > config should be set to false
> > > > > >
> > > > > > From a user's point of view, this seems
> > > > > >
> > > > > > counter-intuitive.
> > > > > >
> > > > > > In
> > > > > >
> > > > > > order to
> > > > > >
> > > > > > auto-create topics the broker's
> > > > > >
> > > > > > auto.create.topics.enable
> > > > > >
> > > > > > config
> > > > > >
> > > > > > should be
> > > > > >
> > > > > > set to false? It seems like the server-side
> > > > > >
> > > > > > auto-create is
> > > > > >
> > > > > > unrelated to
> > > > > >
> > > > > > the client-side auto-create. We could have both turned
> > > > > >
> > > > > > on
> > > > > >
> > > > > > (and
> > > > > >
> > > > > > I'm
> > > > > >
> > > > > > sure
> > > > > >
> > > > > > that in the real world, people will try this
> > > > > >
> > > > > > configuration...)
> > > > > >
> > > > > > There's no
> > > > > >
> > > > > > reason not to support this, I think.
> > > > > >
> > > > > > We should add some documentation explaining the
> > > > > >
> > > > > > difference
> > > > > >
> > > > > > between
> > > > > >
> > > > > > server-side and client-side auto-creation. Without
> > > > > >
> > > > > > documentation,
> > > > > >
> > > > > > an admin
> > > > > >
> > > > > > might think that they had disabled all forms of
> > > > > >
> > > > > > auto-creation by
> > > > > >
> > > > > > setting
> > > > > >
> > > > > > the -side setting to false-- but this is not the case,
> > > > > >
> > > > > > of
> > > > > >
> > > > > > course.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > > > >
> > > > > > Hi Dhruvil,
> > > > > >
> > > > > > Thanks for reading the KIP!
> > > > > > That was the general idea for deprecation. We would
> > > > > >
> > > > > > log a
> > > > > >
> > > > > > warning
> > > > > >
> > > > > > when the
> > > > > >
> > > > > > config is enabled on the broker.
> > > > > > I also believe that there would be a change to
> > > > > >
> > > > > > documentation.
> > > > > >
> > > > > > If there is anything else that should be done, please
> > > > > >
> > > > > > let
> > > > > >
> > > > > > me
> > > > > >
> > > > > > know!
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > > > > >
> > > > > > dhruvil@confluent.io>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > Thanks for the KIP, this is great!
> > > > > >
> > > > > > Could you add some more information about what
> > > > > >
> > > > > > deprecating
> > > > > >
> > > > > > the
> > > > > >
> > > > > > broker
> > > > > >
> > > > > > configuration means? Would we log a warning in the
> > > > > >
> > > > > > logs
> > > > > >
> > > > > > when
> > > > > >
> > > > > > auto
> > > > > >
> > > > > > topic
> > > > > >
> > > > > > creation is enabled on the broker, for example?
> > > > > >
> > > > > > Thanks,
> > > > > > Dhruvil
> > > > > >
> > > > > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > > > > >
> > > > > > jolshan@confluent.io>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > I'd like to start a discussion thread for KIP-487. This KIP plans
> > to
> > > > > > deprecate the current system of
> > > > > >
> > > > > > auto-creating
> > > > > >
> > > > > > topics
> > > > > >
> > > > > > through requests to the metadata and give the
> > > > > >
> > > > > > producer the
> > > > > >
> > > > > > ability to
> > > > > >
> > > > > > automatically create topics instead.
> > > > > >
> > > > > > More information can be found here:
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > > > >
> > > > > > Thank you,
> > > > > > Justine Olshan
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Harsha Ch <ha...@gmail.com>.

Hi Colin,
"Hmm... I'm not sure I follow.  Users don't have to build their own
tooling, right?  They can use any of the shell scripts that we've shipped
in the last few releases.  For example, if any of your users run it, this
shell script will delete all of the topics from your non-security-enabled
cluster:

./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list 2>/dev/null
| xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete
--topic

They will need to fill in the correct bootstrap servers list, of course,
not localhost.  This deletion script will work on some pretty old brokers,
even back to the 0.10 releases.  It seems a little odd to trust your users
with this power, but not trust them to avoid changing a particular
configuration key."

The above will blocked by the server if we set delete.topic.enable to false
and thats exactly what I am asking for.

"The downside is that if we wanted to check a server side configuration
before sending the create topics request, the code would be more complex.
The behavior would also not be consistent with how topic auto-creation is
handled in Kafka Streams."

I am not sure why you need to check server side configuration before
sending create topics request. A user enables producer side config to
create topics.
Producer sends a request to the broker and if the broker has
auto.topic.create.enable to true (default) it will allow creation of
topics. If it set to false it returns error back to the client.
I don't see how this behavior will be different in Kafka streams. By
default server allows the topic creation and with this KIP, It will only
allow creation of topic when both producer and server side are turned on.
Its exactly the same behavior in KIP-361.

"In general, it would be nice if we could keep the security and access
control model simple and not introduce a lot of special cases and
exceptions.  Kafka has basically converged on using ACLs and
CreateTopicPolicy classes to control who can create topics and where.
Adding more knobs seems like a step backwards, especially when the proposed
knobs don't work consistently across components, and don't provide true
security."
This is not about access control at all. Shipping sane defaults should be
prioritized.
We keep talking about CreateTopicPolicy and yet we don't have default one
and asking every user of Kafka implement their own doesn't make sense here.
I am asking about exact behavior that we shipped for consumer side
https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation

I still didn't get any response why this behavior shouldn't be exactly like
Kafka consumer and why do we want to have producer to overrider server side
config and while not allowing consumer to do so.
We are not even allowing the same contract and this will cause more
confusion from the users standpoint.

Thanks,
Harsha



On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe <cm...@apache.org> wrote:

> On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> > Not sure how the AdminClient applies here, It is an external API and
> > is not part of KafkaProducer so any user who updates to latest version of
> > Kafka with this feature get to create the topics.
> > They have to build a tooling around AdminClient allow themselves to
> create
> > topics.
>
> Hi Harsha,
>
> Hmm... I'm not sure I follow.  Users don't have to build their own
> tooling, right?  They can use any of the shell scripts that we've shipped
> in the last few releases.  For example, if any of your users run it, this
> shell script will delete all of the topics from your non-security-enabled
> cluster:
>
> ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list 2>/dev/null
> | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete
> --topic
>
> They will need to fill in the correct bootstrap servers list, of course,
> not localhost.  This deletion script will work on some pretty old brokers,
> even back to the 0.10 releases.  It seems a little odd to trust your users
> with this power, but not trust them to avoid changing a particular
> configuration key.
>
> > There is no behavior in Kafka producer that allowed users to
> > delete the topics or delete the records. So citing them as an
> > example doesn't makes sense in this context.
>
> I think Kafka Streams is relevant here.  After all, it's software that we
> ship as part of the official Kafka release.  And it auto-creates topics
> even when auto.create.topics.enable is set to false on the broker.
>
> I think that auto.create.topics.enable was never intended as a security
> setting to control access.  It was intended as a way to disable the
> broker-side auto-create feature specifically, because there were some
> problems with that specific feature.  Broker-side auto-creation is
> frustrating because it's on by default, and because it can auto-create
> topics even if the producer or consumer didn't explicitly ask for that.
> Neither of those problems applies to this KIP: producers have to
> specifically opt in, and it won't be on by default.  Basically, we think
> that client-side auto-creation is actually a lot better-- hence this KIP :)
>
> > But there is
> > a functionality which allowed creation of topics if they don't exist in
> the
> > cluster and this behavior could be controlled by a config on the server
> > side. Now with this KIP we are allowing producer to make that decision
> > without any gateway on the server via configs. Any user who is not aware
> of
> > such changes
> > can accidentally create these topics and we are essentially removing a
> > config that exists in brokers today to block this accidental creation and
> > allowing clients to take control.
>
> Again, I hope I'm not misinterpreting, but I don't see how this can be
> turned on accidentally.  The user would have to specifically turn this on
> in the producer by setting the configuration key.
>
> >       I still didn't get any positives of not having server side configs?
> > if you want to turn them on and allow any client to create topics set the
> > default to true
> > and allow users who doesn't want to have this feature let them turn them
> > off. It will be the exact behavior as it is today, as far as producer is
> > concerned. I am not
> > understanding why not having server side configs to gateways such a hard
> > requirement and this behavior exists today. As far I am concerned this
> > breaks the backward compatibility.
>
> The downside is that if we wanted to check a server side configuration
> before sending the create topics request, the code would be more complex.
> The behavior would also not be consistent with how topic auto-creation is
> handled in Kafka Streams.
>
> In general, it would be nice if we could keep the security and access
> control model simple and not introduce a lot of special cases and
> exceptions.  Kafka has basically converged on using ACLs and
> CreateTopicPolicy classes to control who can create topics and where.
> Adding more knobs seems like a step backwards, especially when the proposed
> knobs don't work consistently across components, and don't provide true
> security.
>
> Maybe we should support an insecure mode where there are still users and
> ACLs.  That would help people who don't want to set up Kerberos, etc. but
> still want to protect against misconfigured clients or accidents.  Hadoop
> has something like this, and I think it was useful.  NFS also supported
> (supports?) a mode where you just pass whatever user ID you want and the
> system believes you.  These things clearly don't protect against malicious
> users, but they can help set up policies when needed.
>
> best,
> Colin
>
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi Harsha,
> > >
> > > Thanks for taking a look.
> > >
> > > I'm not sure I follow the discussion about AdminClient.
> KafkaAdminClient
> > > has been around for about 2 years at this point as a public class.
> There
> > > are many programs that use it to automatically create topics.  Kafka
> > > Streams does this, for example.  If any of your users run Kafka
> Streams,
> > > they will be auto-creating topics, regardless of what setting you use
> for
> > > auto.create.topics.enable.
> > >
> > > A big part of the annoyance of auto-topic creation right now is that
> it is
> > > on by default.  The new configuration proposed by KIP-487 wouldn't be.
> > > Users would have to explicitly opt in to the new behavior of
> client-side
> > > topic creation.  If you run without security, you're already putting a
> huge
> > > amount of trust in your users.  For example, you trust them not to
> delete
> > > records with the kafka-delete-records.sh command, or delete topics with
> > > kafka-topics.sh.  Trusting them not to set a certain config value seems
> > > minor in comparison, right?
> > >
> > > best,
> > > Colin
> > >
> > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > > > Hi,
> > > >     Even with policies one needs to implement that, so for every
> user who
> > > > doesn't want a producer to create topics or have limits around
> > > partitions &
> > > > replication factor they have to implement a policy.
> > > >       The KIP is changing the behavior , it might not be introducing
> the
> > > > new functionality but it will enable producers to override the create
> > > topic
> > > > config settings on the broker. What I am asking for to provide a
> config
> > > > that will disable auto creation of topics and if its enabled set some
> > > sane
> > > > defaults so that clients can create a topic with in those limits. I
> don't
> > > > see how this not related to this KIP.
> > > >      If the server config options as I suggested doesn't interest
> you at
> > > > least have a default CreateTopicPolices in place.
> > > >        To give an example, In our environment we disable the
> > > > auto.create.topic.enable and force users to go through a centralized
> > > > service as we want capture more details about the topic creation and
> > > > requirements. With this KIP, a producer can create a topic with no
> > > bounds.
> > > >  Another example max.message.size we define that at cluster level
> and one
> > > > can override max.messsage.size at topic level, any misconfiguration
> at
> > > this
> > > > will cause service degradation.  Its not always about the rogue
> clients,
> > > > users can easily misconfigure and can cause an outage.
> > > > Again we can talk about CreateTopicPolicy but without having a
> default
> > > > implementation and asking everyone to implement their own while
> changing
> > > > the behavior in producer  doesn't make sense to me.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > > >
> > > > > Hi Harsha,
> > > > >
> > > > > I mentioned policies and the authorizer. For example, with
> > > > > CreateTopicPolicy, you can implement the limits you describe. If
> you
> > > have
> > > > > ideas of how that should be improved, please submit a KIP. My
> point is
> > > that
> > > > > this KIP is not introducing any new functionality with regards to
> what
> > > > > rogue clients can do. It's using the existing protocol that is
> already
> > > > > exposed via the AdminClient. So, I don't think we need to address
> it in
> > > > > this KIP. Does that make sense?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <
> kafka@harsha.io>
> > > > > wrote:
> > > > >
> > > > > Ismael,
> > > > > Sure AdminClient can do that and we should've shipped a config or
> use
> > > the
> > > > > existing one to block that. Not all users are yet to upgrade to
> > > AdminClient
> > > > > and start using that to cause issues yet. In shared environment we
> > > should
> > > > > allow server to set sane defaults and not allow every client to go
> > > ahead
> > > > > create random no.of topic/partitions and replication factor. Even
> if
> > > the
> > > > > users want to allow topic creation proposed in the KIP , it makes
> > > sense to
> > > > > have some guards against the no.of partitions and replication
> factor.
> > > > > Authorizer is not always an answer to block requests and having
> users
> > > set
> > > > > server side configs to protect a multi-tenant environment is
> required.
> > > In a
> > > > > non-secure environment Authorizer is a blunt instrument either you
> end
> > > up
> > > > > blocking everyone or allowing everyone.
> > > > > I am asking to have server side that allow clients to create
> topics or
> > > not
> > > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > > replication-factor.
> > > > >
> > > > > -Harsha
> > > > >
> > > > > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> > > > >
> > > > > Harsha,
> > > > >
> > > > > Rogue clients can use the admin client to create topics and
> partitions.
> > > > > ACLs and policies can help in that case as well as this one.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > > Thanks for the KIP.
> > > > > "When server-side auto-creation is disabled, client-side
> auto-creation
> > > > > will try to use client-side configurations"
> > > > > If I understand correctly, this KIP is removing any server-side
> > > blocking
> > > > > client auto creation of topic?
> > > > > if so this will present potential issue of rogue client creating
> ton of
> > > > > topic-partitions and potentially bringing down the service for
> everyone
> > > > >
> > > > > or
> > > > >
> > > > > degrade the service itself.
> > > > > By reading the KIP its not clear to me that there is a clear way to
> > > block
> > > > > auto creation topics of all together from clients by server side
> > > config.
> > > > > Server side configs of default topic, partitions should take higher
> > > > > precedence and client shouldn't be able to create a topic with
> higher
> > > > >
> > > > > no.of
> > > > >
> > > > > partitions, replication than what server config specifies.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <
> jolshan@confluent.io>
> > > > > wrote:
> > > > >
> > > > > Hi all,
> > > > > I made some changes to the KIP. Hopefully this configuration change
> > > > >
> > > > > will
> > > > >
> > > > > make things a little clearer.
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > > >
> > > > > Please let me know if you have any feedback or questions!
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I think you bring up a good point. It would be better if we didn't
> ever
> > > > > have to set up client-side configuration for this feature, and
> KIP-464
> > > > > would let us skip this entirely.
> > > > >
> > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > > >
> > > > > Hi Mickael,
> > > > > I agree that KIP-464 works on newer brokers, but I was a bit
> worried
> > > > >
> > > > > how
> > > > >
> > > > > things would play out on older brokers that* do not *have KIP 464
> > > > >
> > > > > included.
> > > > >
> > > > > Is it enough to throw an error in this case when producer configs
> are
> > > > >
> > > > > not
> > > > >
> > > > > specified?
> > > > >
> > > > > I think the right thing to do would be to log an error message in
> the
> > > > > client. We will need to have that capability in any case, to cover
> > > > > scenarios like the client trying to auto-create a topic that they
> don't
> > > > > have permission to create. Or a client trying to create a topic on
> a
> > > > >
> > > > > broker
> > > > >
> > > > > so old that CreateTopicsRequest is not supported.
> > > > >
> > > > > The big downside to relying on KIP-464 is that it is a very recent
> > > > >
> > > > > feature
> > > > >
> > > > > -- so recent that it hasn't even made its way to any official
> Apache
> > > > > release. It's scheduled for the upcoming 2.4 release in a few
> months.
> > > > >
> > > > > So if you view this KIP as a step towards removing broker-side
> > > > > auto-create, you might want to support older brokers just to
> accelerate
> > > > > adoption, and hasten the day when we can finally flip broker-side
> > > > > auto-create to off (or even remove it entirely).
> > > > >
> > > > > I have to agree, though, that having client-side configurations for
> > > > >
> > > > > number
> > > > >
> > > > > of partitions and replication factor is messy. Maybe it would be
> worth
> > > > >
> > > > > it
> > > > >
> > > > > to restrict support to post-KIP-464 brokers, if we could avoid
> creating
> > > > > more configs.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> > > > >
> > > > > mickael.maison@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > We can rely on KIP-464 which allows to omit the partition count or
> > > > > replication factor when creating a topic. In that case, the broker
> > > > >
> > > > > defaults
> > > > >
> > > > > are used.
> > > > >
> > > > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <
> jolshan@confluent.io>
> > > > > wrote:
> > > > >
> > > > > Michael,
> > > > > That makes sense to me!
> > > > > To clarify, in the current state of the KIP, the producer does not
> > > > >
> > > > > rely
> > > > >
> > > > > on
> > > > >
> > > > > the broker to autocreate--if the broker's config is disabled, then
> > > > >
> > > > > the
> > > > >
> > > > > producer can autocreate on its own with a create topic request (the
> > > > >
> > > > > same
> > > > >
> > > > > type of request the admin client uses).
> > > > > However, if both configs are enabled, the broker will autocreate
> > > > >
> > > > > through
> > > > >
> > > > > a
> > > > >
> > > > > metadata request before the producer gets a chance. Of course, the
> way
> > > > >
> > > > > to
> > > > >
> > > > > avoid this, is to do as you suggested, and set
> > > > >
> > > > > the
> > > > >
> > > > > "allow_auto_topic_creation" field to false.
> > > > >
> > > > > I think the only thing we need to be careful with in this setup is
> > > > >
> > > > > without
> > > > >
> > > > > KIP 464, we can not use broker defaults for this topic. A user
> needs
> > > > >
> > > > > to
> > > > >
> > > > > specify the number of partition and replication factor in the
> config.
> > > > >
> > > > > An
> > > > >
> > > > > alternative to this is to have coded defaults for when these
> > > > >
> > > > > configs
> > > > >
> > > > > are
> > > > >
> > > > > unspecified, but it is not immediately apparent what these defaults
> > > > >
> > > > > should
> > > > >
> > > > > be.
> > > > >
> > > > > Thanks again for reading my KIP,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > > > >
> > > > > mickael.maison@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the response!
> > > > > In my opinion, it would be better if the producer did not rely at
> > > > >
> > > > > all
> > > > >
> > > > > on the broker auto create feature as this is what we're aiming to
> > > > > deprecate. When requesting metadata we can set the
> > > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > > creation. Then if the topic is not existing, send a
> CreateTopicRequest.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Currently the way it is implemented, the broker auto-creation
> > > > >
> > > > > configuration
> > > > >
> > > > > takes precedence. The producer will not use the CreateTopics
> > > > >
> > > > > request.
> > > > >
> > > > > (Technically it can--but the topic will already be created
> > > > >
> > > > > through
> > > > >
> > > > > the
> > > > >
> > > > > broker, so it will never try to create the topic.) It is possible
> to
> > > > > change this however, and I'd be happy to
> > > > >
> > > > > discuss
> > > > >
> > > > > the
> > > > >
> > > > > benefits of this alternative.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > > > >
> > > > > mickael.maison@gmail.com>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP!
> > > > >
> > > > > In case auto creation is enabled on both the client and server,
> > > > >
> > > > > will
> > > > >
> > > > > the producer still use the AdminClient (CreateTopics request)
> > > > >
> > > > > to
> > > > >
> > > > > create topics? and not rely on the broker auto create. I'm
> guessing the
> > > > > answer is yes but can you make it explicit.
> > > > >
> > > > > Thank you
> > > > >
> > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi,
> > > > > Just a friendly reminder to take a look at this KIP if you
> > > > >
> > > > > have
> > > > >
> > > > > the
> > > > >
> > > > > time.
> > > > >
> > > > > I was thinking about broker vs. client default precedence,
> > > > >
> > > > > and I
> > > > >
> > > > > think it
> > > > >
> > > > > makes sense to keep the broker as the default used when both
> > > > >
> > > > > client-side
> > > > >
> > > > > and broker-side defaults are configured. The idea is that
> > > > >
> > > > > there
> > > > >
> > > > > would be
> > > > >
> > > > > pretty clear documentation, and that many systems with
> > > > >
> > > > > configurations
> > > > >
> > > > > that
> > > > >
> > > > > the client could not change would likely have the auto-create
> > > > >
> > > > > default
> > > > >
> > > > > off.
> > > > >
> > > > > (In cloud for example).
> > > > >
> > > > > It also seems like in most cases, the consumer config
> > > > > 'allow.auto.create.topics' was created to actually prevent
> > > > >
> > > > > the
> > > > >
> > > > > creation
> > > > >
> > > > > of
> > > > >
> > > > > topics, so the loss of creation functionality will not be a
> > > > >
> > > > > big
> > > > >
> > > > > problem.
> > > > >
> > > > > I'm happy to discuss any other compatibility problems or
> > > > >
> > > > > components
> > > > >
> > > > > of
> > > > >
> > > > > this KIP.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > I was looking at this KIP again, and there is a decision I
> > > > >
> > > > > made
> > > > >
> > > > > that I
> > > > >
> > > > > think is worth discussing.
> > > > >
> > > > > In the case where both the broker and producer's
> > > > > 'auto.create.topics.enable' are set to true, we have to
> > > > >
> > > > > choose
> > > > >
> > > > > either
> > > > >
> > > > > the
> > > > >
> > > > > broker configs or the producer configs for the replication
> > > > > factor/partitions.
> > > > >
> > > > > Currently, the decision is to have the broker defaults take
> > > > >
> > > > > precedence.
> > > > >
> > > > > (It is easier to do this in the implementation.) It also
> > > > >
> > > > > makes
> > > > >
> > > > > some
> > > > >
> > > > > sense
> > > > >
> > > > > for this behavior to take precedence since this behavior
> > > > >
> > > > > already
> > > > >
> > > > > occurs as
> > > > >
> > > > > the default.
> > > > >
> > > > > However, I was wondering if it would be odd for those who
> > > > >
> > > > > can
> > > > >
> > > > > only
> > > > >
> > > > > see
> > > > >
> > > > > the
> > > > >
> > > > > producer side to set configs for replication
> > > > >
> > > > > factor/partitions
> > > > >
> > > > > and
> > > > >
> > > > > see
> > > > >
> > > > > different results. Currently the documentation for the
> > > > >
> > > > > config
> > > > >
> > > > > states
> > > > >
> > > > > that
> > > > >
> > > > > the config values are only used when the broker config is
> > > > >
> > > > > not
> > > > >
> > > > > enabled,
> > > > >
> > > > > but
> > > > >
> > > > > this might not always be clear to the user. Changing the
> > > > >
> > > > > code
> > > > >
> > > > > to
> > > > >
> > > > > have
> > > > >
> > > > > the
> > > > >
> > > > > producer's configurations take precedence is possible, but
> > > > >
> > > > > I
> > > > >
> > > > > was
> > > > >
> > > > > wondering
> > > > >
> > > > > what everyone thought.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Just a quick update--
> > > > >
> > > > > It seems that enabling both the broker and producer
> > > > >
> > > > > configs
> > > > >
> > > > > works
> > > > >
> > > > > fine,
> > > > >
> > > > > except that the broker configurations for partitions,
> > > > >
> > > > > replication
> > > > >
> > > > > factor
> > > > >
> > > > > take precedence.
> > > > > I don't know if that is something we would want to
> > > > >
> > > > > change, but
> > > > >
> > > > > I'll be
> > > > >
> > > > > updating the KIP for now to reflect this. Perhaps we would
> > > > >
> > > > > want to
> > > > >
> > > > > add more
> > > > >
> > > > > to the documentation of the the producer configs to
> > > > >
> > > > > clarify.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Colin,
> > > > >
> > > > > Thanks for looking at the KIP. I can definitely add to
> > > > >
> > > > > the
> > > > >
> > > > > title
> > > > >
> > > > > to
> > > > >
> > > > > make
> > > > >
> > > > > it more clear.
> > > > >
> > > > > It makes sense that both configurations could be turned
> > > > >
> > > > > on
> > > > >
> > > > > since
> > > > >
> > > > > there
> > > > >
> > > > > are many cases where the user can not control the
> > > > >
> > > > > server-side
> > > > >
> > > > > configurations. I was a little concerned about how both
> > > > >
> > > > > interacting
> > > > >
> > > > > would
> > > > >
> > > > > work out -- if there would be to many requests for new
> > > > >
> > > > > topics,
> > > > >
> > > > > for
> > > > >
> > > > > example.
> > > > >
> > > > > But it since it does make sense to allow both
> > > > >
> > > > > configurations
> > > > >
> > > > > enabled, I
> > > > >
> > > > > will test out some scenarios and I'll change the KIP to
> > > > >
> > > > > support
> > > > >
> > > > > this.
> > > > >
> > > > > I also agree with documentation about distinguishing the
> > > > >
> > > > > differences. I
> > > > >
> > > > > was having some trouble with the wording but I like the
> > > > >
> > > > > phrases
> > > > >
> > > > > "server-side" and "client-side." That's a good
> > > > >
> > > > > distinction I
> > > > >
> > > > > can
> > > > >
> > > > > use
> > > > >
> > > > > when
> > > > >
> > > > > describing.
> > > > >
> > > > > I'll try to update the KIP soon keeping everyone's input
> > > > >
> > > > > in
> > > > >
> > > > > mind.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > > > >
> > > > > cmccabe@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP. This seems like a good step towards
> > > > >
> > > > > removing
> > > > >
> > > > > server-side topic auto-creation.
> > > > >
> > > > > We should add included "client-side" to the title of
> > > > >
> > > > > the KIP
> > > > >
> > > > > somewhere,
> > > > >
> > > > > to make it clear that we're talking about client-side
> > > > >
> > > > > auto
> > > > >
> > > > > creation.
> > > > >
> > > > > The KIP says:
> > > > >
> > > > > In order to automatically create topics with the
> > > > >
> > > > > producer, the
> > > > >
> > > > > producer's
> > > > >
> > > > > auto.create.topics.enable config must be set to true
> > > > >
> > > > > and
> > > > >
> > > > > the
> > > > >
> > > > > broker
> > > > >
> > > > > config should be set to false
> > > > >
> > > > > From a user's point of view, this seems
> > > > >
> > > > > counter-intuitive.
> > > > >
> > > > > In
> > > > >
> > > > > order to
> > > > >
> > > > > auto-create topics the broker's
> > > > >
> > > > > auto.create.topics.enable
> > > > >
> > > > > config
> > > > >
> > > > > should be
> > > > >
> > > > > set to false? It seems like the server-side
> > > > >
> > > > > auto-create is
> > > > >
> > > > > unrelated to
> > > > >
> > > > > the client-side auto-create. We could have both turned
> > > > >
> > > > > on
> > > > >
> > > > > (and
> > > > >
> > > > > I'm
> > > > >
> > > > > sure
> > > > >
> > > > > that in the real world, people will try this
> > > > >
> > > > > configuration...)
> > > > >
> > > > > There's no
> > > > >
> > > > > reason not to support this, I think.
> > > > >
> > > > > We should add some documentation explaining the
> > > > >
> > > > > difference
> > > > >
> > > > > between
> > > > >
> > > > > server-side and client-side auto-creation. Without
> > > > >
> > > > > documentation,
> > > > >
> > > > > an admin
> > > > >
> > > > > might think that they had disabled all forms of
> > > > >
> > > > > auto-creation by
> > > > >
> > > > > setting
> > > > >
> > > > > the -side setting to false-- but this is not the case,
> > > > >
> > > > > of
> > > > >
> > > > > course.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > > >
> > > > > Hi Dhruvil,
> > > > >
> > > > > Thanks for reading the KIP!
> > > > > That was the general idea for deprecation. We would
> > > > >
> > > > > log a
> > > > >
> > > > > warning
> > > > >
> > > > > when the
> > > > >
> > > > > config is enabled on the broker.
> > > > > I also believe that there would be a change to
> > > > >
> > > > > documentation.
> > > > >
> > > > > If there is anything else that should be done, please
> > > > >
> > > > > let
> > > > >
> > > > > me
> > > > >
> > > > > know!
> > > > >
> > > > > Justine
> > > > >
> > > > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > > > >
> > > > > dhruvil@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP, this is great!
> > > > >
> > > > > Could you add some more information about what
> > > > >
> > > > > deprecating
> > > > >
> > > > > the
> > > > >
> > > > > broker
> > > > >
> > > > > configuration means? Would we log a warning in the
> > > > >
> > > > > logs
> > > > >
> > > > > when
> > > > >
> > > > > auto
> > > > >
> > > > > topic
> > > > >
> > > > > creation is enabled on the broker, for example?
> > > > >
> > > > > Thanks,
> > > > > Dhruvil
> > > > >
> > > > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > > > >
> > > > > jolshan@confluent.io>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > I'd like to start a discussion thread for KIP-487. This KIP plans
> to
> > > > > deprecate the current system of
> > > > >
> > > > > auto-creating
> > > > >
> > > > > topics
> > > > >
> > > > > through requests to the metadata and give the
> > > > >
> > > > > producer the
> > > > >
> > > > > ability to
> > > > >
> > > > > automatically create topics instead.
> > > > >
> > > > > More information can be found here:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > > >
> > > > > Thank you,
> > > > > Justine Olshan
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Colin McCabe <cm...@apache.org>.

On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote:
> Not sure how the AdminClient applies here, It is an external API and
> is not part of KafkaProducer so any user who updates to latest version of
> Kafka with this feature get to create the topics.
> They have to build a tooling around AdminClient allow themselves to create
> topics.

Hi Harsha,

Hmm... I'm not sure I follow.  Users don't have to build their own tooling, right?  They can use any of the shell scripts that we've shipped in the last few releases.  For example, if any of your users run it, this shell script will delete all of the topics from your non-security-enabled cluster:

./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list 2>/dev/null | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic

They will need to fill in the correct bootstrap servers list, of course, not localhost.  This deletion script will work on some pretty old brokers, even back to the 0.10 releases.  It seems a little odd to trust your users with this power, but not trust them to avoid changing a particular configuration key.

> There is no behavior in Kafka producer that allowed users to
> delete the topics or delete the records. So citing them as an
> example doesn't makes sense in this context.

I think Kafka Streams is relevant here.  After all, it's software that we ship as part of the official Kafka release.  And it auto-creates topics even when auto.create.topics.enable is set to false on the broker.

I think that auto.create.topics.enable was never intended as a security setting to control access.  It was intended as a way to disable the broker-side auto-create feature specifically, because there were some problems with that specific feature.  Broker-side auto-creation is frustrating because it's on by default, and because it can auto-create topics even if the producer or consumer didn't explicitly ask for that.  Neither of those problems applies to this KIP: producers have to specifically opt in, and it won't be on by default.  Basically, we think that client-side auto-creation is actually a lot better-- hence this KIP :)

> But there is
> a functionality which allowed creation of topics if they don't exist in the
> cluster and this behavior could be controlled by a config on the server
> side. Now with this KIP we are allowing producer to make that decision
> without any gateway on the server via configs. Any user who is not aware of
> such changes
> can accidentally create these topics and we are essentially removing a
> config that exists in brokers today to block this accidental creation and
> allowing clients to take control.

Again, I hope I'm not misinterpreting, but I don't see how this can be turned on accidentally.  The user would have to specifically turn this on in the producer by setting the configuration key.

>       I still didn't get any positives of not having server side configs?
> if you want to turn them on and allow any client to create topics set the
> default to true
> and allow users who doesn't want to have this feature let them turn them
> off. It will be the exact behavior as it is today, as far as producer is
> concerned. I am not
> understanding why not having server side configs to gateways such a hard
> requirement and this behavior exists today. As far I am concerned this
> breaks the backward compatibility.

The downside is that if we wanted to check a server side configuration before sending the create topics request, the code would be more complex.  The behavior would also not be consistent with how topic auto-creation is handled in Kafka Streams.

In general, it would be nice if we could keep the security and access control model simple and not introduce a lot of special cases and exceptions.  Kafka has basically converged on using ACLs and CreateTopicPolicy classes to control who can create topics and where.  Adding more knobs seems like a step backwards, especially when the proposed knobs don't work consistently across components, and don't provide true security.

Maybe we should support an insecure mode where there are still users and ACLs.  That would help people who don't want to set up Kerberos, etc. but still want to protect against misconfigured clients or accidents.  Hadoop has something like this, and I think it was useful.  NFS also supported (supports?) a mode where you just pass whatever user ID you want and the system believes you.  These things clearly don't protect against malicious users, but they can help set up policies when needed.

best,
Colin

> 
> Thanks,
> Harsha
> 
> 
> On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe <cm...@apache.org> wrote:
> 
> > Hi Harsha,
> >
> > Thanks for taking a look.
> >
> > I'm not sure I follow the discussion about AdminClient.  KafkaAdminClient
> > has been around for about 2 years at this point as a public class.  There
> > are many programs that use it to automatically create topics.  Kafka
> > Streams does this, for example.  If any of your users run Kafka Streams,
> > they will be auto-creating topics, regardless of what setting you use for
> > auto.create.topics.enable.
> >
> > A big part of the annoyance of auto-topic creation right now is that it is
> > on by default.  The new configuration proposed by KIP-487 wouldn't be.
> > Users would have to explicitly opt in to the new behavior of client-side
> > topic creation.  If you run without security, you're already putting a huge
> > amount of trust in your users.  For example, you trust them not to delete
> > records with the kafka-delete-records.sh command, or delete topics with
> > kafka-topics.sh.  Trusting them not to set a certain config value seems
> > minor in comparison, right?
> >
> > best,
> > Colin
> >
> > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > > Hi,
> > >     Even with policies one needs to implement that, so for every user who
> > > doesn't want a producer to create topics or have limits around
> > partitions &
> > > replication factor they have to implement a policy.
> > >       The KIP is changing the behavior , it might not be introducing the
> > > new functionality but it will enable producers to override the create
> > topic
> > > config settings on the broker. What I am asking for to provide a config
> > > that will disable auto creation of topics and if its enabled set some
> > sane
> > > defaults so that clients can create a topic with in those limits. I don't
> > > see how this not related to this KIP.
> > >      If the server config options as I suggested doesn't interest you at
> > > least have a default CreateTopicPolices in place.
> > >        To give an example, In our environment we disable the
> > > auto.create.topic.enable and force users to go through a centralized
> > > service as we want capture more details about the topic creation and
> > > requirements. With this KIP, a producer can create a topic with no
> > bounds.
> > >  Another example max.message.size we define that at cluster level and one
> > > can override max.messsage.size at topic level, any misconfiguration at
> > this
> > > will cause service degradation.  Its not always about the rogue clients,
> > > users can easily misconfigure and can cause an outage.
> > > Again we can talk about CreateTopicPolicy but without having a default
> > > implementation and asking everyone to implement their own while changing
> > > the behavior in producer  doesn't make sense to me.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi Harsha,
> > > >
> > > > I mentioned policies and the authorizer. For example, with
> > > > CreateTopicPolicy, you can implement the limits you describe. If you
> > have
> > > > ideas of how that should be improved, please submit a KIP. My point is
> > that
> > > > this KIP is not introducing any new functionality with regards to what
> > > > rogue clients can do. It's using the existing protocol that is already
> > > > exposed via the AdminClient. So, I don't think we need to address it in
> > > > this KIP. Does that make sense?
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io>
> > > > wrote:
> > > >
> > > > Ismael,
> > > > Sure AdminClient can do that and we should've shipped a config or use
> > the
> > > > existing one to block that. Not all users are yet to upgrade to
> > AdminClient
> > > > and start using that to cause issues yet. In shared environment we
> > should
> > > > allow server to set sane defaults and not allow every client to go
> > ahead
> > > > create random no.of topic/partitions and replication factor. Even if
> > the
> > > > users want to allow topic creation proposed in the KIP , it makes
> > sense to
> > > > have some guards against the no.of partitions and replication factor.
> > > > Authorizer is not always an answer to block requests and having users
> > set
> > > > server side configs to protect a multi-tenant environment is required.
> > In a
> > > > non-secure environment Authorizer is a blunt instrument either you end
> > up
> > > > blocking everyone or allowing everyone.
> > > > I am asking to have server side that allow clients to create topics or
> > not
> > > > , if they are allowed set a ceiling on max no.of partitions and
> > > > replication-factor.
> > > >
> > > > -Harsha
> > > >
> > > > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> > > >
> > > > Harsha,
> > > >
> > > > Rogue clients can use the admin client to create topics and partitions.
> > > > ACLs and policies can help in that case as well as this one.
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > > Thanks for the KIP.
> > > > "When server-side auto-creation is disabled, client-side auto-creation
> > > > will try to use client-side configurations"
> > > > If I understand correctly, this KIP is removing any server-side
> > blocking
> > > > client auto creation of topic?
> > > > if so this will present potential issue of rogue client creating ton of
> > > > topic-partitions and potentially bringing down the service for everyone
> > > >
> > > > or
> > > >
> > > > degrade the service itself.
> > > > By reading the KIP its not clear to me that there is a clear way to
> > block
> > > > auto creation topics of all together from clients by server side
> > config.
> > > > Server side configs of default topic, partitions should take higher
> > > > precedence and client shouldn't be able to create a topic with higher
> > > >
> > > > no.of
> > > >
> > > > partitions, replication than what server config specifies.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> > > > wrote:
> > > >
> > > > Hi all,
> > > > I made some changes to the KIP. Hopefully this configuration change
> > > >
> > > > will
> > > >
> > > > make things a little clearer.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > > >
> > > > Please let me know if you have any feedback or questions!
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> > > >
> > > > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think you bring up a good point. It would be better if we didn't ever
> > > > have to set up client-side configuration for this feature, and KIP-464
> > > > would let us skip this entirely.
> > > >
> > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > > >
> > > > Hi Mickael,
> > > > I agree that KIP-464 works on newer brokers, but I was a bit worried
> > > >
> > > > how
> > > >
> > > > things would play out on older brokers that* do not *have KIP 464
> > > >
> > > > included.
> > > >
> > > > Is it enough to throw an error in this case when producer configs are
> > > >
> > > > not
> > > >
> > > > specified?
> > > >
> > > > I think the right thing to do would be to log an error message in the
> > > > client. We will need to have that capability in any case, to cover
> > > > scenarios like the client trying to auto-create a topic that they don't
> > > > have permission to create. Or a client trying to create a topic on a
> > > >
> > > > broker
> > > >
> > > > so old that CreateTopicsRequest is not supported.
> > > >
> > > > The big downside to relying on KIP-464 is that it is a very recent
> > > >
> > > > feature
> > > >
> > > > -- so recent that it hasn't even made its way to any official Apache
> > > > release. It's scheduled for the upcoming 2.4 release in a few months.
> > > >
> > > > So if you view this KIP as a step towards removing broker-side
> > > > auto-create, you might want to support older brokers just to accelerate
> > > > adoption, and hasten the day when we can finally flip broker-side
> > > > auto-create to off (or even remove it entirely).
> > > >
> > > > I have to agree, though, that having client-side configurations for
> > > >
> > > > number
> > > >
> > > > of partitions and replication factor is messy. Maybe it would be worth
> > > >
> > > > it
> > > >
> > > > to restrict support to post-KIP-464 brokers, if we could avoid creating
> > > > more configs.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> > > >
> > > > mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > We can rely on KIP-464 which allows to omit the partition count or
> > > > replication factor when creating a topic. In that case, the broker
> > > >
> > > > defaults
> > > >
> > > > are used.
> > > >
> > > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > > > wrote:
> > > >
> > > > Michael,
> > > > That makes sense to me!
> > > > To clarify, in the current state of the KIP, the producer does not
> > > >
> > > > rely
> > > >
> > > > on
> > > >
> > > > the broker to autocreate--if the broker's config is disabled, then
> > > >
> > > > the
> > > >
> > > > producer can autocreate on its own with a create topic request (the
> > > >
> > > > same
> > > >
> > > > type of request the admin client uses).
> > > > However, if both configs are enabled, the broker will autocreate
> > > >
> > > > through
> > > >
> > > > a
> > > >
> > > > metadata request before the producer gets a chance. Of course, the way
> > > >
> > > > to
> > > >
> > > > avoid this, is to do as you suggested, and set
> > > >
> > > > the
> > > >
> > > > "allow_auto_topic_creation" field to false.
> > > >
> > > > I think the only thing we need to be careful with in this setup is
> > > >
> > > > without
> > > >
> > > > KIP 464, we can not use broker defaults for this topic. A user needs
> > > >
> > > > to
> > > >
> > > > specify the number of partition and replication factor in the config.
> > > >
> > > > An
> > > >
> > > > alternative to this is to have coded defaults for when these
> > > >
> > > > configs
> > > >
> > > > are
> > > >
> > > > unspecified, but it is not immediately apparent what these defaults
> > > >
> > > > should
> > > >
> > > > be.
> > > >
> > > > Thanks again for reading my KIP,
> > > > Justine
> > > >
> > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > > >
> > > > mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the response!
> > > > In my opinion, it would be better if the producer did not rely at
> > > >
> > > > all
> > > >
> > > > on the broker auto create feature as this is what we're aiming to
> > > > deprecate. When requesting metadata we can set the
> > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > creation. Then if the topic is not existing, send a CreateTopicRequest.
> > > >
> > > > What do you think?
> > > >
> > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Currently the way it is implemented, the broker auto-creation
> > > >
> > > > configuration
> > > >
> > > > takes precedence. The producer will not use the CreateTopics
> > > >
> > > > request.
> > > >
> > > > (Technically it can--but the topic will already be created
> > > >
> > > > through
> > > >
> > > > the
> > > >
> > > > broker, so it will never try to create the topic.) It is possible to
> > > > change this however, and I'd be happy to
> > > >
> > > > discuss
> > > >
> > > > the
> > > >
> > > > benefits of this alternative.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > > >
> > > > mickael.maison@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > In case auto creation is enabled on both the client and server,
> > > >
> > > > will
> > > >
> > > > the producer still use the AdminClient (CreateTopics request)
> > > >
> > > > to
> > > >
> > > > create topics? and not rely on the broker auto create. I'm guessing the
> > > > answer is yes but can you make it explicit.
> > > >
> > > > Thank you
> > > >
> > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Hi,
> > > > Just a friendly reminder to take a look at this KIP if you
> > > >
> > > > have
> > > >
> > > > the
> > > >
> > > > time.
> > > >
> > > > I was thinking about broker vs. client default precedence,
> > > >
> > > > and I
> > > >
> > > > think it
> > > >
> > > > makes sense to keep the broker as the default used when both
> > > >
> > > > client-side
> > > >
> > > > and broker-side defaults are configured. The idea is that
> > > >
> > > > there
> > > >
> > > > would be
> > > >
> > > > pretty clear documentation, and that many systems with
> > > >
> > > > configurations
> > > >
> > > > that
> > > >
> > > > the client could not change would likely have the auto-create
> > > >
> > > > default
> > > >
> > > > off.
> > > >
> > > > (In cloud for example).
> > > >
> > > > It also seems like in most cases, the consumer config
> > > > 'allow.auto.create.topics' was created to actually prevent
> > > >
> > > > the
> > > >
> > > > creation
> > > >
> > > > of
> > > >
> > > > topics, so the loss of creation functionality will not be a
> > > >
> > > > big
> > > >
> > > > problem.
> > > >
> > > > I'm happy to discuss any other compatibility problems or
> > > >
> > > > components
> > > >
> > > > of
> > > >
> > > > this KIP.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > > >
> > > > jolshan@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I was looking at this KIP again, and there is a decision I
> > > >
> > > > made
> > > >
> > > > that I
> > > >
> > > > think is worth discussing.
> > > >
> > > > In the case where both the broker and producer's
> > > > 'auto.create.topics.enable' are set to true, we have to
> > > >
> > > > choose
> > > >
> > > > either
> > > >
> > > > the
> > > >
> > > > broker configs or the producer configs for the replication
> > > > factor/partitions.
> > > >
> > > > Currently, the decision is to have the broker defaults take
> > > >
> > > > precedence.
> > > >
> > > > (It is easier to do this in the implementation.) It also
> > > >
> > > > makes
> > > >
> > > > some
> > > >
> > > > sense
> > > >
> > > > for this behavior to take precedence since this behavior
> > > >
> > > > already
> > > >
> > > > occurs as
> > > >
> > > > the default.
> > > >
> > > > However, I was wondering if it would be odd for those who
> > > >
> > > > can
> > > >
> > > > only
> > > >
> > > > see
> > > >
> > > > the
> > > >
> > > > producer side to set configs for replication
> > > >
> > > > factor/partitions
> > > >
> > > > and
> > > >
> > > > see
> > > >
> > > > different results. Currently the documentation for the
> > > >
> > > > config
> > > >
> > > > states
> > > >
> > > > that
> > > >
> > > > the config values are only used when the broker config is
> > > >
> > > > not
> > > >
> > > > enabled,
> > > >
> > > > but
> > > >
> > > > this might not always be clear to the user. Changing the
> > > >
> > > > code
> > > >
> > > > to
> > > >
> > > > have
> > > >
> > > > the
> > > >
> > > > producer's configurations take precedence is possible, but
> > > >
> > > > I
> > > >
> > > > was
> > > >
> > > > wondering
> > > >
> > > > what everyone thought.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Just a quick update--
> > > >
> > > > It seems that enabling both the broker and producer
> > > >
> > > > configs
> > > >
> > > > works
> > > >
> > > > fine,
> > > >
> > > > except that the broker configurations for partitions,
> > > >
> > > > replication
> > > >
> > > > factor
> > > >
> > > > take precedence.
> > > > I don't know if that is something we would want to
> > > >
> > > > change, but
> > > >
> > > > I'll be
> > > >
> > > > updating the KIP for now to reflect this. Perhaps we would
> > > >
> > > > want to
> > > >
> > > > add more
> > > >
> > > > to the documentation of the the producer configs to
> > > >
> > > > clarify.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Hi Colin,
> > > >
> > > > Thanks for looking at the KIP. I can definitely add to
> > > >
> > > > the
> > > >
> > > > title
> > > >
> > > > to
> > > >
> > > > make
> > > >
> > > > it more clear.
> > > >
> > > > It makes sense that both configurations could be turned
> > > >
> > > > on
> > > >
> > > > since
> > > >
> > > > there
> > > >
> > > > are many cases where the user can not control the
> > > >
> > > > server-side
> > > >
> > > > configurations. I was a little concerned about how both
> > > >
> > > > interacting
> > > >
> > > > would
> > > >
> > > > work out -- if there would be to many requests for new
> > > >
> > > > topics,
> > > >
> > > > for
> > > >
> > > > example.
> > > >
> > > > But it since it does make sense to allow both
> > > >
> > > > configurations
> > > >
> > > > enabled, I
> > > >
> > > > will test out some scenarios and I'll change the KIP to
> > > >
> > > > support
> > > >
> > > > this.
> > > >
> > > > I also agree with documentation about distinguishing the
> > > >
> > > > differences. I
> > > >
> > > > was having some trouble with the wording but I like the
> > > >
> > > > phrases
> > > >
> > > > "server-side" and "client-side." That's a good
> > > >
> > > > distinction I
> > > >
> > > > can
> > > >
> > > > use
> > > >
> > > > when
> > > >
> > > > describing.
> > > >
> > > > I'll try to update the KIP soon keeping everyone's input
> > > >
> > > > in
> > > >
> > > > mind.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > > >
> > > > cmccabe@apache.org
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP. This seems like a good step towards
> > > >
> > > > removing
> > > >
> > > > server-side topic auto-creation.
> > > >
> > > > We should add included "client-side" to the title of
> > > >
> > > > the KIP
> > > >
> > > > somewhere,
> > > >
> > > > to make it clear that we're talking about client-side
> > > >
> > > > auto
> > > >
> > > > creation.
> > > >
> > > > The KIP says:
> > > >
> > > > In order to automatically create topics with the
> > > >
> > > > producer, the
> > > >
> > > > producer's
> > > >
> > > > auto.create.topics.enable config must be set to true
> > > >
> > > > and
> > > >
> > > > the
> > > >
> > > > broker
> > > >
> > > > config should be set to false
> > > >
> > > > From a user's point of view, this seems
> > > >
> > > > counter-intuitive.
> > > >
> > > > In
> > > >
> > > > order to
> > > >
> > > > auto-create topics the broker's
> > > >
> > > > auto.create.topics.enable
> > > >
> > > > config
> > > >
> > > > should be
> > > >
> > > > set to false? It seems like the server-side
> > > >
> > > > auto-create is
> > > >
> > > > unrelated to
> > > >
> > > > the client-side auto-create. We could have both turned
> > > >
> > > > on
> > > >
> > > > (and
> > > >
> > > > I'm
> > > >
> > > > sure
> > > >
> > > > that in the real world, people will try this
> > > >
> > > > configuration...)
> > > >
> > > > There's no
> > > >
> > > > reason not to support this, I think.
> > > >
> > > > We should add some documentation explaining the
> > > >
> > > > difference
> > > >
> > > > between
> > > >
> > > > server-side and client-side auto-creation. Without
> > > >
> > > > documentation,
> > > >
> > > > an admin
> > > >
> > > > might think that they had disabled all forms of
> > > >
> > > > auto-creation by
> > > >
> > > > setting
> > > >
> > > > the -side setting to false-- but this is not the case,
> > > >
> > > > of
> > > >
> > > > course.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > > >
> > > > Hi Dhruvil,
> > > >
> > > > Thanks for reading the KIP!
> > > > That was the general idea for deprecation. We would
> > > >
> > > > log a
> > > >
> > > > warning
> > > >
> > > > when the
> > > >
> > > > config is enabled on the broker.
> > > > I also believe that there would be a change to
> > > >
> > > > documentation.
> > > >
> > > > If there is anything else that should be done, please
> > > >
> > > > let
> > > >
> > > > me
> > > >
> > > > know!
> > > >
> > > > Justine
> > > >
> > > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > > >
> > > > dhruvil@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP, this is great!
> > > >
> > > > Could you add some more information about what
> > > >
> > > > deprecating
> > > >
> > > > the
> > > >
> > > > broker
> > > >
> > > > configuration means? Would we log a warning in the
> > > >
> > > > logs
> > > >
> > > > when
> > > >
> > > > auto
> > > >
> > > > topic
> > > >
> > > > creation is enabled on the broker, for example?
> > > >
> > > > Thanks,
> > > > Dhruvil
> > > >
> > > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > > >
> > > > jolshan@confluent.io>
> > > >
> > > > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > > > deprecate the current system of
> > > >
> > > > auto-creating
> > > >
> > > > topics
> > > >
> > > > through requests to the metadata and give the
> > > >
> > > > producer the
> > > >
> > > > ability to
> > > >
> > > > automatically create topics instead.
> > > >
> > > > More information can be found here:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > > >
> > > > Thank you,
> > > > Justine Olshan
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Harsha Ch <ha...@gmail.com>.

Hi Colin,
         There is no behavior in Kafka producer that allowed users to
delete the topics or delete the records. So
citing them as an example doesn't makes sense in this context. But there is
a functionality which allowed creation of topics if they don't exist in the
cluster and this behavior could be controlled by a config on the server
side. Now with this KIP we are allowing producer to make that decision
without any gateway on the server via configs. Any user who is not aware of
such changes
can accidentally create these topics and we are essentially removing a
config that exists in brokers today to block this accidental creation and
allowing clients to take control.
      Not sure how the AdminClient applies here, It is an external API and
is not part of KafkaProducer so any user who updates to latest version of
Kafka with this feature get to create the topics.
They have to build a tooling around AdminClient allow themselves to create
topics.
      I still didn't get any positives of not having server side configs?
if you want to turn them on and allow any client to create topics set the
default to true
and allow users who doesn't want to have this feature let them turn them
off. It will be the exact behavior as it is today, as far as producer is
concerned. I am not
understanding why not having server side configs to gateways such a hard
requirement and this behavior exists today. As far I am concerned this
breaks the backward compatibility.

Thanks,
Harsha


On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe <cm...@apache.org> wrote:

> Hi Harsha,
>
> Thanks for taking a look.
>
> I'm not sure I follow the discussion about AdminClient.  KafkaAdminClient
> has been around for about 2 years at this point as a public class.  There
> are many programs that use it to automatically create topics.  Kafka
> Streams does this, for example.  If any of your users run Kafka Streams,
> they will be auto-creating topics, regardless of what setting you use for
> auto.create.topics.enable.
>
> A big part of the annoyance of auto-topic creation right now is that it is
> on by default.  The new configuration proposed by KIP-487 wouldn't be.
> Users would have to explicitly opt in to the new behavior of client-side
> topic creation.  If you run without security, you're already putting a huge
> amount of trust in your users.  For example, you trust them not to delete
> records with the kafka-delete-records.sh command, or delete topics with
> kafka-topics.sh.  Trusting them not to set a certain config value seems
> minor in comparison, right?
>
> best,
> Colin
>
> On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> > Hi,
> >     Even with policies one needs to implement that, so for every user who
> > doesn't want a producer to create topics or have limits around
> partitions &
> > replication factor they have to implement a policy.
> >       The KIP is changing the behavior , it might not be introducing the
> > new functionality but it will enable producers to override the create
> topic
> > config settings on the broker. What I am asking for to provide a config
> > that will disable auto creation of topics and if its enabled set some
> sane
> > defaults so that clients can create a topic with in those limits. I don't
> > see how this not related to this KIP.
> >      If the server config options as I suggested doesn't interest you at
> > least have a default CreateTopicPolices in place.
> >        To give an example, In our environment we disable the
> > auto.create.topic.enable and force users to go through a centralized
> > service as we want capture more details about the topic creation and
> > requirements. With this KIP, a producer can create a topic with no
> bounds.
> >  Another example max.message.size we define that at cluster level and one
> > can override max.messsage.size at topic level, any misconfiguration at
> this
> > will cause service degradation.  Its not always about the rogue clients,
> > users can easily misconfigure and can cause an outage.
> > Again we can talk about CreateTopicPolicy but without having a default
> > implementation and asking everyone to implement their own while changing
> > the behavior in producer  doesn't make sense to me.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Harsha,
> > >
> > > I mentioned policies and the authorizer. For example, with
> > > CreateTopicPolicy, you can implement the limits you describe. If you
> have
> > > ideas of how that should be improved, please submit a KIP. My point is
> that
> > > this KIP is not introducing any new functionality with regards to what
> > > rogue clients can do. It's using the existing protocol that is already
> > > exposed via the AdminClient. So, I don't think we need to address it in
> > > this KIP. Does that make sense?
> > >
> > > Ismael
> > >
> > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > Ismael,
> > > Sure AdminClient can do that and we should've shipped a config or use
> the
> > > existing one to block that. Not all users are yet to upgrade to
> AdminClient
> > > and start using that to cause issues yet. In shared environment we
> should
> > > allow server to set sane defaults and not allow every client to go
> ahead
> > > create random no.of topic/partitions and replication factor. Even if
> the
> > > users want to allow topic creation proposed in the KIP , it makes
> sense to
> > > have some guards against the no.of partitions and replication factor.
> > > Authorizer is not always an answer to block requests and having users
> set
> > > server side configs to protect a multi-tenant environment is required.
> In a
> > > non-secure environment Authorizer is a blunt instrument either you end
> up
> > > blocking everyone or allowing everyone.
> > > I am asking to have server side that allow clients to create topics or
> not
> > > , if they are allowed set a ceiling on max no.of partitions and
> > > replication-factor.
> > >
> > > -Harsha
> > >
> > > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> > >
> > > Harsha,
> > >
> > > Rogue clients can use the admin client to create topics and partitions.
> > > ACLs and policies can help in that case as well as this one.
> > >
> > > Ismael
> > >
> > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > > Thanks for the KIP.
> > > "When server-side auto-creation is disabled, client-side auto-creation
> > > will try to use client-side configurations"
> > > If I understand correctly, this KIP is removing any server-side
> blocking
> > > client auto creation of topic?
> > > if so this will present potential issue of rogue client creating ton of
> > > topic-partitions and potentially bringing down the service for everyone
> > >
> > > or
> > >
> > > degrade the service itself.
> > > By reading the KIP its not clear to me that there is a clear way to
> block
> > > auto creation topics of all together from clients by server side
> config.
> > > Server side configs of default topic, partitions should take higher
> > > precedence and client shouldn't be able to create a topic with higher
> > >
> > > no.of
> > >
> > > partitions, replication than what server config specifies.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> > > wrote:
> > >
> > > Hi all,
> > > I made some changes to the KIP. Hopefully this configuration change
> > >
> > > will
> > >
> > > make things a little clearer.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > >
> > > Please let me know if you have any feedback or questions!
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think you bring up a good point. It would be better if we didn't ever
> > > have to set up client-side configuration for this feature, and KIP-464
> > > would let us skip this entirely.
> > >
> > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > >
> > > Hi Mickael,
> > > I agree that KIP-464 works on newer brokers, but I was a bit worried
> > >
> > > how
> > >
> > > things would play out on older brokers that* do not *have KIP 464
> > >
> > > included.
> > >
> > > Is it enough to throw an error in this case when producer configs are
> > >
> > > not
> > >
> > > specified?
> > >
> > > I think the right thing to do would be to log an error message in the
> > > client. We will need to have that capability in any case, to cover
> > > scenarios like the client trying to auto-create a topic that they don't
> > > have permission to create. Or a client trying to create a topic on a
> > >
> > > broker
> > >
> > > so old that CreateTopicsRequest is not supported.
> > >
> > > The big downside to relying on KIP-464 is that it is a very recent
> > >
> > > feature
> > >
> > > -- so recent that it hasn't even made its way to any official Apache
> > > release. It's scheduled for the upcoming 2.4 release in a few months.
> > >
> > > So if you view this KIP as a step towards removing broker-side
> > > auto-create, you might want to support older brokers just to accelerate
> > > adoption, and hasten the day when we can finally flip broker-side
> > > auto-create to off (or even remove it entirely).
> > >
> > > I have to agree, though, that having client-side configurations for
> > >
> > > number
> > >
> > > of partitions and replication factor is messy. Maybe it would be worth
> > >
> > > it
> > >
> > > to restrict support to post-KIP-464 brokers, if we could avoid creating
> > > more configs.
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > We can rely on KIP-464 which allows to omit the partition count or
> > > replication factor when creating a topic. In that case, the broker
> > >
> > > defaults
> > >
> > > are used.
> > >
> > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > > wrote:
> > >
> > > Michael,
> > > That makes sense to me!
> > > To clarify, in the current state of the KIP, the producer does not
> > >
> > > rely
> > >
> > > on
> > >
> > > the broker to autocreate--if the broker's config is disabled, then
> > >
> > > the
> > >
> > > producer can autocreate on its own with a create topic request (the
> > >
> > > same
> > >
> > > type of request the admin client uses).
> > > However, if both configs are enabled, the broker will autocreate
> > >
> > > through
> > >
> > > a
> > >
> > > metadata request before the producer gets a chance. Of course, the way
> > >
> > > to
> > >
> > > avoid this, is to do as you suggested, and set
> > >
> > > the
> > >
> > > "allow_auto_topic_creation" field to false.
> > >
> > > I think the only thing we need to be careful with in this setup is
> > >
> > > without
> > >
> > > KIP 464, we can not use broker defaults for this topic. A user needs
> > >
> > > to
> > >
> > > specify the number of partition and replication factor in the config.
> > >
> > > An
> > >
> > > alternative to this is to have coded defaults for when these
> > >
> > > configs
> > >
> > > are
> > >
> > > unspecified, but it is not immediately apparent what these defaults
> > >
> > > should
> > >
> > > be.
> > >
> > > Thanks again for reading my KIP,
> > > Justine
> > >
> > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the response!
> > > In my opinion, it would be better if the producer did not rely at
> > >
> > > all
> > >
> > > on the broker auto create feature as this is what we're aiming to
> > > deprecate. When requesting metadata we can set the
> > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > creation. Then if the topic is not existing, send a CreateTopicRequest.
> > >
> > > What do you think?
> > >
> > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Currently the way it is implemented, the broker auto-creation
> > >
> > > configuration
> > >
> > > takes precedence. The producer will not use the CreateTopics
> > >
> > > request.
> > >
> > > (Technically it can--but the topic will already be created
> > >
> > > through
> > >
> > > the
> > >
> > > broker, so it will never try to create the topic.) It is possible to
> > > change this however, and I'd be happy to
> > >
> > > discuss
> > >
> > > the
> > >
> > > benefits of this alternative.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP!
> > >
> > > In case auto creation is enabled on both the client and server,
> > >
> > > will
> > >
> > > the producer still use the AdminClient (CreateTopics request)
> > >
> > > to
> > >
> > > create topics? and not rely on the broker auto create. I'm guessing the
> > > answer is yes but can you make it explicit.
> > >
> > > Thank you
> > >
> > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi,
> > > Just a friendly reminder to take a look at this KIP if you
> > >
> > > have
> > >
> > > the
> > >
> > > time.
> > >
> > > I was thinking about broker vs. client default precedence,
> > >
> > > and I
> > >
> > > think it
> > >
> > > makes sense to keep the broker as the default used when both
> > >
> > > client-side
> > >
> > > and broker-side defaults are configured. The idea is that
> > >
> > > there
> > >
> > > would be
> > >
> > > pretty clear documentation, and that many systems with
> > >
> > > configurations
> > >
> > > that
> > >
> > > the client could not change would likely have the auto-create
> > >
> > > default
> > >
> > > off.
> > >
> > > (In cloud for example).
> > >
> > > It also seems like in most cases, the consumer config
> > > 'allow.auto.create.topics' was created to actually prevent
> > >
> > > the
> > >
> > > creation
> > >
> > > of
> > >
> > > topics, so the loss of creation functionality will not be a
> > >
> > > big
> > >
> > > problem.
> > >
> > > I'm happy to discuss any other compatibility problems or
> > >
> > > components
> > >
> > > of
> > >
> > > this KIP.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > >
> > > jolshan@confluent.io
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I was looking at this KIP again, and there is a decision I
> > >
> > > made
> > >
> > > that I
> > >
> > > think is worth discussing.
> > >
> > > In the case where both the broker and producer's
> > > 'auto.create.topics.enable' are set to true, we have to
> > >
> > > choose
> > >
> > > either
> > >
> > > the
> > >
> > > broker configs or the producer configs for the replication
> > > factor/partitions.
> > >
> > > Currently, the decision is to have the broker defaults take
> > >
> > > precedence.
> > >
> > > (It is easier to do this in the implementation.) It also
> > >
> > > makes
> > >
> > > some
> > >
> > > sense
> > >
> > > for this behavior to take precedence since this behavior
> > >
> > > already
> > >
> > > occurs as
> > >
> > > the default.
> > >
> > > However, I was wondering if it would be odd for those who
> > >
> > > can
> > >
> > > only
> > >
> > > see
> > >
> > > the
> > >
> > > producer side to set configs for replication
> > >
> > > factor/partitions
> > >
> > > and
> > >
> > > see
> > >
> > > different results. Currently the documentation for the
> > >
> > > config
> > >
> > > states
> > >
> > > that
> > >
> > > the config values are only used when the broker config is
> > >
> > > not
> > >
> > > enabled,
> > >
> > > but
> > >
> > > this might not always be clear to the user. Changing the
> > >
> > > code
> > >
> > > to
> > >
> > > have
> > >
> > > the
> > >
> > > producer's configurations take precedence is possible, but
> > >
> > > I
> > >
> > > was
> > >
> > > wondering
> > >
> > > what everyone thought.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Just a quick update--
> > >
> > > It seems that enabling both the broker and producer
> > >
> > > configs
> > >
> > > works
> > >
> > > fine,
> > >
> > > except that the broker configurations for partitions,
> > >
> > > replication
> > >
> > > factor
> > >
> > > take precedence.
> > > I don't know if that is something we would want to
> > >
> > > change, but
> > >
> > > I'll be
> > >
> > > updating the KIP for now to reflect this. Perhaps we would
> > >
> > > want to
> > >
> > > add more
> > >
> > > to the documentation of the the producer configs to
> > >
> > > clarify.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for looking at the KIP. I can definitely add to
> > >
> > > the
> > >
> > > title
> > >
> > > to
> > >
> > > make
> > >
> > > it more clear.
> > >
> > > It makes sense that both configurations could be turned
> > >
> > > on
> > >
> > > since
> > >
> > > there
> > >
> > > are many cases where the user can not control the
> > >
> > > server-side
> > >
> > > configurations. I was a little concerned about how both
> > >
> > > interacting
> > >
> > > would
> > >
> > > work out -- if there would be to many requests for new
> > >
> > > topics,
> > >
> > > for
> > >
> > > example.
> > >
> > > But it since it does make sense to allow both
> > >
> > > configurations
> > >
> > > enabled, I
> > >
> > > will test out some scenarios and I'll change the KIP to
> > >
> > > support
> > >
> > > this.
> > >
> > > I also agree with documentation about distinguishing the
> > >
> > > differences. I
> > >
> > > was having some trouble with the wording but I like the
> > >
> > > phrases
> > >
> > > "server-side" and "client-side." That's a good
> > >
> > > distinction I
> > >
> > > can
> > >
> > > use
> > >
> > > when
> > >
> > > describing.
> > >
> > > I'll try to update the KIP soon keeping everyone's input
> > >
> > > in
> > >
> > > mind.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > >
> > > cmccabe@apache.org
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP. This seems like a good step towards
> > >
> > > removing
> > >
> > > server-side topic auto-creation.
> > >
> > > We should add included "client-side" to the title of
> > >
> > > the KIP
> > >
> > > somewhere,
> > >
> > > to make it clear that we're talking about client-side
> > >
> > > auto
> > >
> > > creation.
> > >
> > > The KIP says:
> > >
> > > In order to automatically create topics with the
> > >
> > > producer, the
> > >
> > > producer's
> > >
> > > auto.create.topics.enable config must be set to true
> > >
> > > and
> > >
> > > the
> > >
> > > broker
> > >
> > > config should be set to false
> > >
> > > From a user's point of view, this seems
> > >
> > > counter-intuitive.
> > >
> > > In
> > >
> > > order to
> > >
> > > auto-create topics the broker's
> > >
> > > auto.create.topics.enable
> > >
> > > config
> > >
> > > should be
> > >
> > > set to false? It seems like the server-side
> > >
> > > auto-create is
> > >
> > > unrelated to
> > >
> > > the client-side auto-create. We could have both turned
> > >
> > > on
> > >
> > > (and
> > >
> > > I'm
> > >
> > > sure
> > >
> > > that in the real world, people will try this
> > >
> > > configuration...)
> > >
> > > There's no
> > >
> > > reason not to support this, I think.
> > >
> > > We should add some documentation explaining the
> > >
> > > difference
> > >
> > > between
> > >
> > > server-side and client-side auto-creation. Without
> > >
> > > documentation,
> > >
> > > an admin
> > >
> > > might think that they had disabled all forms of
> > >
> > > auto-creation by
> > >
> > > setting
> > >
> > > the -side setting to false-- but this is not the case,
> > >
> > > of
> > >
> > > course.
> > >
> > > best,
> > > Colin
> > >
> > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > >
> > > Hi Dhruvil,
> > >
> > > Thanks for reading the KIP!
> > > That was the general idea for deprecation. We would
> > >
> > > log a
> > >
> > > warning
> > >
> > > when the
> > >
> > > config is enabled on the broker.
> > > I also believe that there would be a change to
> > >
> > > documentation.
> > >
> > > If there is anything else that should be done, please
> > >
> > > let
> > >
> > > me
> > >
> > > know!
> > >
> > > Justine
> > >
> > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > >
> > > dhruvil@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP, this is great!
> > >
> > > Could you add some more information about what
> > >
> > > deprecating
> > >
> > > the
> > >
> > > broker
> > >
> > > configuration means? Would we log a warning in the
> > >
> > > logs
> > >
> > > when
> > >
> > > auto
> > >
> > > topic
> > >
> > > creation is enabled on the broker, for example?
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > > deprecate the current system of
> > >
> > > auto-creating
> > >
> > > topics
> > >
> > > through requests to the metadata and give the
> > >
> > > producer the
> > >
> > > ability to
> > >
> > > automatically create topics instead.
> > >
> > > More information can be found here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > >
> > > Thank you,
> > > Justine Olshan
> > >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Colin McCabe <cm...@apache.org>.

Hi Harsha,

Thanks for taking a look.

I'm not sure I follow the discussion about AdminClient.  KafkaAdminClient has been around for about 2 years at this point as a public class.  There are many programs that use it to automatically create topics.  Kafka Streams does this, for example.  If any of your users run Kafka Streams, they will be auto-creating topics, regardless of what setting you use for auto.create.topics.enable.

A big part of the annoyance of auto-topic creation right now is that it is on by default.  The new configuration proposed by KIP-487 wouldn't be.  Users would have to explicitly opt in to the new behavior of client-side topic creation.  If you run without security, you're already putting a huge amount of trust in your users.  For example, you trust them not to delete records with the kafka-delete-records.sh command, or delete topics with kafka-topics.sh.  Trusting them not to set a certain config value seems minor in comparison, right?

best,
Colin

On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote:
> Hi,
>     Even with policies one needs to implement that, so for every user who
> doesn't want a producer to create topics or have limits around partitions &
> replication factor they have to implement a policy.
>       The KIP is changing the behavior , it might not be introducing the
> new functionality but it will enable producers to override the create topic
> config settings on the broker. What I am asking for to provide a config
> that will disable auto creation of topics and if its enabled set some sane
> defaults so that clients can create a topic with in those limits. I don't
> see how this not related to this KIP.
>      If the server config options as I suggested doesn't interest you at
> least have a default CreateTopicPolices in place.
>        To give an example, In our environment we disable the
> auto.create.topic.enable and force users to go through a centralized
> service as we want capture more details about the topic creation and
> requirements. With this KIP, a producer can create a topic with no bounds.
>  Another example max.message.size we define that at cluster level and one
> can override max.messsage.size at topic level, any misconfiguration at this
> will cause service degradation.  Its not always about the rogue clients,
> users can easily misconfigure and can cause an outage.
> Again we can talk about CreateTopicPolicy but without having a default
> implementation and asking everyone to implement their own while changing
> the behavior in producer  doesn't make sense to me.
> 
> Thanks,
> Harsha
> 
> 
> On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk> wrote:
> 
> > Hi Harsha,
> >
> > I mentioned policies and the authorizer. For example, with
> > CreateTopicPolicy, you can implement the limits you describe. If you have
> > ideas of how that should be improved, please submit a KIP. My point is that
> > this KIP is not introducing any new functionality with regards to what
> > rogue clients can do. It's using the existing protocol that is already
> > exposed via the AdminClient. So, I don't think we need to address it in
> > this KIP. Does that make sense?
> >
> > Ismael
> >
> > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > Ismael,
> > Sure AdminClient can do that and we should've shipped a config or use the
> > existing one to block that. Not all users are yet to upgrade to AdminClient
> > and start using that to cause issues yet. In shared environment we should
> > allow server to set sane defaults and not allow every client to go ahead
> > create random no.of topic/partitions and replication factor. Even if the
> > users want to allow topic creation proposed in the KIP , it makes sense to
> > have some guards against the no.of partitions and replication factor.
> > Authorizer is not always an answer to block requests and having users set
> > server side configs to protect a multi-tenant environment is required. In a
> > non-secure environment Authorizer is a blunt instrument either you end up
> > blocking everyone or allowing everyone.
> > I am asking to have server side that allow clients to create topics or not
> > , if they are allowed set a ceiling on max no.of partitions and
> > replication-factor.
> >
> > -Harsha
> >
> > On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
> >
> > Harsha,
> >
> > Rogue clients can use the admin client to create topics and partitions.
> > ACLs and policies can help in that case as well as this one.
> >
> > Ismael
> >
> > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> >
> > wrote:
> >
> > Hi Justine,
> > Thanks for the KIP.
> > "When server-side auto-creation is disabled, client-side auto-creation
> > will try to use client-side configurations"
> > If I understand correctly, this KIP is removing any server-side blocking
> > client auto creation of topic?
> > if so this will present potential issue of rogue client creating ton of
> > topic-partitions and potentially bringing down the service for everyone
> >
> > or
> >
> > degrade the service itself.
> > By reading the KIP its not clear to me that there is a clear way to block
> > auto creation topics of all together from clients by server side config.
> > Server side configs of default topic, partitions should take higher
> > precedence and client shouldn't be able to create a topic with higher
> >
> > no.of
> >
> > partitions, replication than what server config specifies.
> >
> > Thanks,
> > Harsha
> >
> > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> > wrote:
> >
> > Hi all,
> > I made some changes to the KIP. Hopefully this configuration change
> >
> > will
> >
> > make things a little clearer.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> >
> > Please let me know if you have any feedback or questions!
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> >
> > wrote:
> >
> > Hi Mickael,
> >
> > I think you bring up a good point. It would be better if we didn't ever
> > have to set up client-side configuration for this feature, and KIP-464
> > would let us skip this entirely.
> >
> > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> >
> > Hi Mickael,
> > I agree that KIP-464 works on newer brokers, but I was a bit worried
> >
> > how
> >
> > things would play out on older brokers that* do not *have KIP 464
> >
> > included.
> >
> > Is it enough to throw an error in this case when producer configs are
> >
> > not
> >
> > specified?
> >
> > I think the right thing to do would be to log an error message in the
> > client. We will need to have that capability in any case, to cover
> > scenarios like the client trying to auto-create a topic that they don't
> > have permission to create. Or a client trying to create a topic on a
> >
> > broker
> >
> > so old that CreateTopicsRequest is not supported.
> >
> > The big downside to relying on KIP-464 is that it is a very recent
> >
> > feature
> >
> > -- so recent that it hasn't even made its way to any official Apache
> > release. It's scheduled for the upcoming 2.4 release in a few months.
> >
> > So if you view this KIP as a step towards removing broker-side
> > auto-create, you might want to support older brokers just to accelerate
> > adoption, and hasten the day when we can finally flip broker-side
> > auto-create to off (or even remove it entirely).
> >
> > I have to agree, though, that having client-side configurations for
> >
> > number
> >
> > of partitions and replication factor is messy. Maybe it would be worth
> >
> > it
> >
> > to restrict support to post-KIP-464 brokers, if we could avoid creating
> > more configs.
> >
> > best,
> > Colin
> >
> > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> >
> > mickael.maison@gmail.com
> >
> > wrote:
> >
> > Hi Justine,
> >
> > We can rely on KIP-464 which allows to omit the partition count or
> > replication factor when creating a topic. In that case, the broker
> >
> > defaults
> >
> > are used.
> >
> > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > wrote:
> >
> > Michael,
> > That makes sense to me!
> > To clarify, in the current state of the KIP, the producer does not
> >
> > rely
> >
> > on
> >
> > the broker to autocreate--if the broker's config is disabled, then
> >
> > the
> >
> > producer can autocreate on its own with a create topic request (the
> >
> > same
> >
> > type of request the admin client uses).
> > However, if both configs are enabled, the broker will autocreate
> >
> > through
> >
> > a
> >
> > metadata request before the producer gets a chance. Of course, the way
> >
> > to
> >
> > avoid this, is to do as you suggested, and set
> >
> > the
> >
> > "allow_auto_topic_creation" field to false.
> >
> > I think the only thing we need to be careful with in this setup is
> >
> > without
> >
> > KIP 464, we can not use broker defaults for this topic. A user needs
> >
> > to
> >
> > specify the number of partition and replication factor in the config.
> >
> > An
> >
> > alternative to this is to have coded defaults for when these
> >
> > configs
> >
> > are
> >
> > unspecified, but it is not immediately apparent what these defaults
> >
> > should
> >
> > be.
> >
> > Thanks again for reading my KIP,
> > Justine
> >
> > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> >
> > mickael.maison@gmail.com
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the response!
> > In my opinion, it would be better if the producer did not rely at
> >
> > all
> >
> > on the broker auto create feature as this is what we're aiming to
> > deprecate. When requesting metadata we can set the
> > "allow_auto_topic_creation" field to false to avoid the broker auto
> > creation. Then if the topic is not existing, send a CreateTopicRequest.
> >
> > What do you think?
> >
> > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Currently the way it is implemented, the broker auto-creation
> >
> > configuration
> >
> > takes precedence. The producer will not use the CreateTopics
> >
> > request.
> >
> > (Technically it can--but the topic will already be created
> >
> > through
> >
> > the
> >
> > broker, so it will never try to create the topic.) It is possible to
> > change this however, and I'd be happy to
> >
> > discuss
> >
> > the
> >
> > benefits of this alternative.
> >
> > Thank you,
> > Justine
> >
> > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> >
> > mickael.maison@gmail.com>
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP!
> >
> > In case auto creation is enabled on both the client and server,
> >
> > will
> >
> > the producer still use the AdminClient (CreateTopics request)
> >
> > to
> >
> > create topics? and not rely on the broker auto create. I'm guessing the
> > answer is yes but can you make it explicit.
> >
> > Thank you
> >
> > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hi,
> > Just a friendly reminder to take a look at this KIP if you
> >
> > have
> >
> > the
> >
> > time.
> >
> > I was thinking about broker vs. client default precedence,
> >
> > and I
> >
> > think it
> >
> > makes sense to keep the broker as the default used when both
> >
> > client-side
> >
> > and broker-side defaults are configured. The idea is that
> >
> > there
> >
> > would be
> >
> > pretty clear documentation, and that many systems with
> >
> > configurations
> >
> > that
> >
> > the client could not change would likely have the auto-create
> >
> > default
> >
> > off.
> >
> > (In cloud for example).
> >
> > It also seems like in most cases, the consumer config
> > 'allow.auto.create.topics' was created to actually prevent
> >
> > the
> >
> > creation
> >
> > of
> >
> > topics, so the loss of creation functionality will not be a
> >
> > big
> >
> > problem.
> >
> > I'm happy to discuss any other compatibility problems or
> >
> > components
> >
> > of
> >
> > this KIP.
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> >
> > jolshan@confluent.io
> >
> > wrote:
> >
> > Hello all,
> >
> > I was looking at this KIP again, and there is a decision I
> >
> > made
> >
> > that I
> >
> > think is worth discussing.
> >
> > In the case where both the broker and producer's
> > 'auto.create.topics.enable' are set to true, we have to
> >
> > choose
> >
> > either
> >
> > the
> >
> > broker configs or the producer configs for the replication
> > factor/partitions.
> >
> > Currently, the decision is to have the broker defaults take
> >
> > precedence.
> >
> > (It is easier to do this in the implementation.) It also
> >
> > makes
> >
> > some
> >
> > sense
> >
> > for this behavior to take precedence since this behavior
> >
> > already
> >
> > occurs as
> >
> > the default.
> >
> > However, I was wondering if it would be odd for those who
> >
> > can
> >
> > only
> >
> > see
> >
> > the
> >
> > producer side to set configs for replication
> >
> > factor/partitions
> >
> > and
> >
> > see
> >
> > different results. Currently the documentation for the
> >
> > config
> >
> > states
> >
> > that
> >
> > the config values are only used when the broker config is
> >
> > not
> >
> > enabled,
> >
> > but
> >
> > this might not always be clear to the user. Changing the
> >
> > code
> >
> > to
> >
> > have
> >
> > the
> >
> > producer's configurations take precedence is possible, but
> >
> > I
> >
> > was
> >
> > wondering
> >
> > what everyone thought.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Just a quick update--
> >
> > It seems that enabling both the broker and producer
> >
> > configs
> >
> > works
> >
> > fine,
> >
> > except that the broker configurations for partitions,
> >
> > replication
> >
> > factor
> >
> > take precedence.
> > I don't know if that is something we would want to
> >
> > change, but
> >
> > I'll be
> >
> > updating the KIP for now to reflect this. Perhaps we would
> >
> > want to
> >
> > add more
> >
> > to the documentation of the the producer configs to
> >
> > clarify.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hi Colin,
> >
> > Thanks for looking at the KIP. I can definitely add to
> >
> > the
> >
> > title
> >
> > to
> >
> > make
> >
> > it more clear.
> >
> > It makes sense that both configurations could be turned
> >
> > on
> >
> > since
> >
> > there
> >
> > are many cases where the user can not control the
> >
> > server-side
> >
> > configurations. I was a little concerned about how both
> >
> > interacting
> >
> > would
> >
> > work out -- if there would be to many requests for new
> >
> > topics,
> >
> > for
> >
> > example.
> >
> > But it since it does make sense to allow both
> >
> > configurations
> >
> > enabled, I
> >
> > will test out some scenarios and I'll change the KIP to
> >
> > support
> >
> > this.
> >
> > I also agree with documentation about distinguishing the
> >
> > differences. I
> >
> > was having some trouble with the wording but I like the
> >
> > phrases
> >
> > "server-side" and "client-side." That's a good
> >
> > distinction I
> >
> > can
> >
> > use
> >
> > when
> >
> > describing.
> >
> > I'll try to update the KIP soon keeping everyone's input
> >
> > in
> >
> > mind.
> >
> > Thanks,
> > Justine
> >
> > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> >
> > cmccabe@apache.org
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP. This seems like a good step towards
> >
> > removing
> >
> > server-side topic auto-creation.
> >
> > We should add included "client-side" to the title of
> >
> > the KIP
> >
> > somewhere,
> >
> > to make it clear that we're talking about client-side
> >
> > auto
> >
> > creation.
> >
> > The KIP says:
> >
> > In order to automatically create topics with the
> >
> > producer, the
> >
> > producer's
> >
> > auto.create.topics.enable config must be set to true
> >
> > and
> >
> > the
> >
> > broker
> >
> > config should be set to false
> >
> > From a user's point of view, this seems
> >
> > counter-intuitive.
> >
> > In
> >
> > order to
> >
> > auto-create topics the broker's
> >
> > auto.create.topics.enable
> >
> > config
> >
> > should be
> >
> > set to false? It seems like the server-side
> >
> > auto-create is
> >
> > unrelated to
> >
> > the client-side auto-create. We could have both turned
> >
> > on
> >
> > (and
> >
> > I'm
> >
> > sure
> >
> > that in the real world, people will try this
> >
> > configuration...)
> >
> > There's no
> >
> > reason not to support this, I think.
> >
> > We should add some documentation explaining the
> >
> > difference
> >
> > between
> >
> > server-side and client-side auto-creation. Without
> >
> > documentation,
> >
> > an admin
> >
> > might think that they had disabled all forms of
> >
> > auto-creation by
> >
> > setting
> >
> > the -side setting to false-- but this is not the case,
> >
> > of
> >
> > course.
> >
> > best,
> > Colin
> >
> > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> >
> > Hi Dhruvil,
> >
> > Thanks for reading the KIP!
> > That was the general idea for deprecation. We would
> >
> > log a
> >
> > warning
> >
> > when the
> >
> > config is enabled on the broker.
> > I also believe that there would be a change to
> >
> > documentation.
> >
> > If there is anything else that should be done, please
> >
> > let
> >
> > me
> >
> > know!
> >
> > Justine
> >
> > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> >
> > dhruvil@confluent.io>
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP, this is great!
> >
> > Could you add some more information about what
> >
> > deprecating
> >
> > the
> >
> > broker
> >
> > configuration means? Would we log a warning in the
> >
> > logs
> >
> > when
> >
> > auto
> >
> > topic
> >
> > creation is enabled on the broker, for example?
> >
> > Thanks,
> > Dhruvil
> >
> > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hello all,
> >
> > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > deprecate the current system of
> >
> > auto-creating
> >
> > topics
> >
> > through requests to the metadata and give the
> >
> > producer the
> >
> > ability to
> >
> > automatically create topics instead.
> >
> > More information can be found here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> >
> > Thank you,
> > Justine Olshan
> >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Harsha Chintalapani <ka...@harsha.io>.

Hi,
    Even with policies one needs to implement that, so for every user who
doesn't want a producer to create topics or have limits around partitions &
replication factor they have to implement a policy.
      The KIP is changing the behavior , it might not be introducing the
new functionality but it will enable producers to override the create topic
config settings on the broker. What I am asking for to provide a config
that will disable auto creation of topics and if its enabled set some sane
defaults so that clients can create a topic with in those limits. I don't
see how this not related to this KIP.
     If the server config options as I suggested doesn't interest you at
least have a default CreateTopicPolices in place.
       To give an example, In our environment we disable the
auto.create.topic.enable and force users to go through a centralized
service as we want capture more details about the topic creation and
requirements. With this KIP, a producer can create a topic with no bounds.
 Another example max.message.size we define that at cluster level and one
can override max.messsage.size at topic level, any misconfiguration at this
will cause service degradation.  Its not always about the rogue clients,
users can easily misconfigure and can cause an outage.
Again we can talk about CreateTopicPolicy but without having a default
implementation and asking everyone to implement their own while changing
the behavior in producer  doesn't make sense to me.

Thanks,
Harsha


On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <is...@juma.me.uk> wrote:

> Hi Harsha,
>
> I mentioned policies and the authorizer. For example, with
> CreateTopicPolicy, you can implement the limits you describe. If you have
> ideas of how that should be improved, please submit a KIP. My point is that
> this KIP is not introducing any new functionality with regards to what
> rogue clients can do. It's using the existing protocol that is already
> exposed via the AdminClient. So, I don't think we need to address it in
> this KIP. Does that make sense?
>
> Ismael
>
> On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> Ismael,
> Sure AdminClient can do that and we should've shipped a config or use the
> existing one to block that. Not all users are yet to upgrade to AdminClient
> and start using that to cause issues yet. In shared environment we should
> allow server to set sane defaults and not allow every client to go ahead
> create random no.of topic/partitions and replication factor. Even if the
> users want to allow topic creation proposed in the KIP , it makes sense to
> have some guards against the no.of partitions and replication factor.
> Authorizer is not always an answer to block requests and having users set
> server side configs to protect a multi-tenant environment is required. In a
> non-secure environment Authorizer is a blunt instrument either you end up
> blocking everyone or allowing everyone.
> I am asking to have server side that allow clients to create topics or not
> , if they are allowed set a ceiling on max no.of partitions and
> replication-factor.
>
> -Harsha
>
> On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
>
> Harsha,
>
> Rogue clients can use the admin client to create topics and partitions.
> ACLs and policies can help in that case as well as this one.
>
> Ismael
>
> On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
>
> wrote:
>
> Hi Justine,
> Thanks for the KIP.
> "When server-side auto-creation is disabled, client-side auto-creation
> will try to use client-side configurations"
> If I understand correctly, this KIP is removing any server-side blocking
> client auto creation of topic?
> if so this will present potential issue of rogue client creating ton of
> topic-partitions and potentially bringing down the service for everyone
>
> or
>
> degrade the service itself.
> By reading the KIP its not clear to me that there is a clear way to block
> auto creation topics of all together from clients by server side config.
> Server side configs of default topic, partitions should take higher
> precedence and client shouldn't be able to create a topic with higher
>
> no.of
>
> partitions, replication than what server config specifies.
>
> Thanks,
> Harsha
>
> On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> wrote:
>
> Hi all,
> I made some changes to the KIP. Hopefully this configuration change
>
> will
>
> make things a little clearer.
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
>
> Please let me know if you have any feedback or questions!
>
> Thank you,
> Justine
>
> On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
>
> wrote:
>
> Hi Mickael,
>
> I think you bring up a good point. It would be better if we didn't ever
> have to set up client-side configuration for this feature, and KIP-464
> would let us skip this entirely.
>
> On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
>
> Hi Mickael,
> I agree that KIP-464 works on newer brokers, but I was a bit worried
>
> how
>
> things would play out on older brokers that* do not *have KIP 464
>
> included.
>
> Is it enough to throw an error in this case when producer configs are
>
> not
>
> specified?
>
> I think the right thing to do would be to log an error message in the
> client. We will need to have that capability in any case, to cover
> scenarios like the client trying to auto-create a topic that they don't
> have permission to create. Or a client trying to create a topic on a
>
> broker
>
> so old that CreateTopicsRequest is not supported.
>
> The big downside to relying on KIP-464 is that it is a very recent
>
> feature
>
> -- so recent that it hasn't even made its way to any official Apache
> release. It's scheduled for the upcoming 2.4 release in a few months.
>
> So if you view this KIP as a step towards removing broker-side
> auto-create, you might want to support older brokers just to accelerate
> adoption, and hasten the day when we can finally flip broker-side
> auto-create to off (or even remove it entirely).
>
> I have to agree, though, that having client-side configurations for
>
> number
>
> of partitions and replication factor is messy. Maybe it would be worth
>
> it
>
> to restrict support to post-KIP-464 brokers, if we could avoid creating
> more configs.
>
> best,
> Colin
>
> On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
>
> mickael.maison@gmail.com
>
> wrote:
>
> Hi Justine,
>
> We can rely on KIP-464 which allows to omit the partition count or
> replication factor when creating a topic. In that case, the broker
>
> defaults
>
> are used.
>
> On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> wrote:
>
> Michael,
> That makes sense to me!
> To clarify, in the current state of the KIP, the producer does not
>
> rely
>
> on
>
> the broker to autocreate--if the broker's config is disabled, then
>
> the
>
> producer can autocreate on its own with a create topic request (the
>
> same
>
> type of request the admin client uses).
> However, if both configs are enabled, the broker will autocreate
>
> through
>
> a
>
> metadata request before the producer gets a chance. Of course, the way
>
> to
>
> avoid this, is to do as you suggested, and set
>
> the
>
> "allow_auto_topic_creation" field to false.
>
> I think the only thing we need to be careful with in this setup is
>
> without
>
> KIP 464, we can not use broker defaults for this topic. A user needs
>
> to
>
> specify the number of partition and replication factor in the config.
>
> An
>
> alternative to this is to have coded defaults for when these
>
> configs
>
> are
>
> unspecified, but it is not immediately apparent what these defaults
>
> should
>
> be.
>
> Thanks again for reading my KIP,
> Justine
>
> On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
>
> mickael.maison@gmail.com
>
> wrote:
>
> Hi Justine,
>
> Thanks for the response!
> In my opinion, it would be better if the producer did not rely at
>
> all
>
> on the broker auto create feature as this is what we're aiming to
> deprecate. When requesting metadata we can set the
> "allow_auto_topic_creation" field to false to avoid the broker auto
> creation. Then if the topic is not existing, send a CreateTopicRequest.
>
> What do you think?
>
> On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Currently the way it is implemented, the broker auto-creation
>
> configuration
>
> takes precedence. The producer will not use the CreateTopics
>
> request.
>
> (Technically it can--but the topic will already be created
>
> through
>
> the
>
> broker, so it will never try to create the topic.) It is possible to
> change this however, and I'd be happy to
>
> discuss
>
> the
>
> benefits of this alternative.
>
> Thank you,
> Justine
>
> On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
>
> mickael.maison@gmail.com>
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP!
>
> In case auto creation is enabled on both the client and server,
>
> will
>
> the producer still use the AdminClient (CreateTopics request)
>
> to
>
> create topics? and not rely on the broker auto create. I'm guessing the
> answer is yes but can you make it explicit.
>
> Thank you
>
> On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Hi,
> Just a friendly reminder to take a look at this KIP if you
>
> have
>
> the
>
> time.
>
> I was thinking about broker vs. client default precedence,
>
> and I
>
> think it
>
> makes sense to keep the broker as the default used when both
>
> client-side
>
> and broker-side defaults are configured. The idea is that
>
> there
>
> would be
>
> pretty clear documentation, and that many systems with
>
> configurations
>
> that
>
> the client could not change would likely have the auto-create
>
> default
>
> off.
>
> (In cloud for example).
>
> It also seems like in most cases, the consumer config
> 'allow.auto.create.topics' was created to actually prevent
>
> the
>
> creation
>
> of
>
> topics, so the loss of creation functionality will not be a
>
> big
>
> problem.
>
> I'm happy to discuss any other compatibility problems or
>
> components
>
> of
>
> this KIP.
>
> Thank you,
> Justine
>
> On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
>
> jolshan@confluent.io
>
> wrote:
>
> Hello all,
>
> I was looking at this KIP again, and there is a decision I
>
> made
>
> that I
>
> think is worth discussing.
>
> In the case where both the broker and producer's
> 'auto.create.topics.enable' are set to true, we have to
>
> choose
>
> either
>
> the
>
> broker configs or the producer configs for the replication
> factor/partitions.
>
> Currently, the decision is to have the broker defaults take
>
> precedence.
>
> (It is easier to do this in the implementation.) It also
>
> makes
>
> some
>
> sense
>
> for this behavior to take precedence since this behavior
>
> already
>
> occurs as
>
> the default.
>
> However, I was wondering if it would be odd for those who
>
> can
>
> only
>
> see
>
> the
>
> producer side to set configs for replication
>
> factor/partitions
>
> and
>
> see
>
> different results. Currently the documentation for the
>
> config
>
> states
>
> that
>
> the config values are only used when the broker config is
>
> not
>
> enabled,
>
> but
>
> this might not always be clear to the user. Changing the
>
> code
>
> to
>
> have
>
> the
>
> producer's configurations take precedence is possible, but
>
> I
>
> was
>
> wondering
>
> what everyone thought.
>
> Thank you,
> Justine
>
> On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Just a quick update--
>
> It seems that enabling both the broker and producer
>
> configs
>
> works
>
> fine,
>
> except that the broker configurations for partitions,
>
> replication
>
> factor
>
> take precedence.
> I don't know if that is something we would want to
>
> change, but
>
> I'll be
>
> updating the KIP for now to reflect this. Perhaps we would
>
> want to
>
> add more
>
> to the documentation of the the producer configs to
>
> clarify.
>
> Thank you,
> Justine
>
> On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Hi Colin,
>
> Thanks for looking at the KIP. I can definitely add to
>
> the
>
> title
>
> to
>
> make
>
> it more clear.
>
> It makes sense that both configurations could be turned
>
> on
>
> since
>
> there
>
> are many cases where the user can not control the
>
> server-side
>
> configurations. I was a little concerned about how both
>
> interacting
>
> would
>
> work out -- if there would be to many requests for new
>
> topics,
>
> for
>
> example.
>
> But it since it does make sense to allow both
>
> configurations
>
> enabled, I
>
> will test out some scenarios and I'll change the KIP to
>
> support
>
> this.
>
> I also agree with documentation about distinguishing the
>
> differences. I
>
> was having some trouble with the wording but I like the
>
> phrases
>
> "server-side" and "client-side." That's a good
>
> distinction I
>
> can
>
> use
>
> when
>
> describing.
>
> I'll try to update the KIP soon keeping everyone's input
>
> in
>
> mind.
>
> Thanks,
> Justine
>
> On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
>
> cmccabe@apache.org
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP. This seems like a good step towards
>
> removing
>
> server-side topic auto-creation.
>
> We should add included "client-side" to the title of
>
> the KIP
>
> somewhere,
>
> to make it clear that we're talking about client-side
>
> auto
>
> creation.
>
> The KIP says:
>
> In order to automatically create topics with the
>
> producer, the
>
> producer's
>
> auto.create.topics.enable config must be set to true
>
> and
>
> the
>
> broker
>
> config should be set to false
>
> From a user's point of view, this seems
>
> counter-intuitive.
>
> In
>
> order to
>
> auto-create topics the broker's
>
> auto.create.topics.enable
>
> config
>
> should be
>
> set to false? It seems like the server-side
>
> auto-create is
>
> unrelated to
>
> the client-side auto-create. We could have both turned
>
> on
>
> (and
>
> I'm
>
> sure
>
> that in the real world, people will try this
>
> configuration...)
>
> There's no
>
> reason not to support this, I think.
>
> We should add some documentation explaining the
>
> difference
>
> between
>
> server-side and client-side auto-creation. Without
>
> documentation,
>
> an admin
>
> might think that they had disabled all forms of
>
> auto-creation by
>
> setting
>
> the -side setting to false-- but this is not the case,
>
> of
>
> course.
>
> best,
> Colin
>
> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
>
> Hi Dhruvil,
>
> Thanks for reading the KIP!
> That was the general idea for deprecation. We would
>
> log a
>
> warning
>
> when the
>
> config is enabled on the broker.
> I also believe that there would be a change to
>
> documentation.
>
> If there is anything else that should be done, please
>
> let
>
> me
>
> know!
>
> Justine
>
> On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
>
> dhruvil@confluent.io>
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP, this is great!
>
> Could you add some more information about what
>
> deprecating
>
> the
>
> broker
>
> configuration means? Would we log a warning in the
>
> logs
>
> when
>
> auto
>
> topic
>
> creation is enabled on the broker, for example?
>
> Thanks,
> Dhruvil
>
> On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Hello all,
>
> I'd like to start a discussion thread for KIP-487. This KIP plans to
> deprecate the current system of
>
> auto-creating
>
> topics
>
> through requests to the metadata and give the
>
> producer the
>
> ability to
>
> automatically create topics instead.
>
> More information can be found here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-487%3A+Automatic+Topic+Creation+on+Producer
>
> Thank you,
> Justine Olshan
>
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Ismael Juma <is...@juma.me.uk>.

Hi Harsha,

I mentioned policies and the authorizer. For example, with
CreateTopicPolicy, you can implement the limits you describe. If you have
ideas of how that should be improved, please submit a KIP. My point is that
this KIP is not introducing any new functionality with regards to what
rogue clients can do. It's using the existing protocol that is already
exposed via the AdminClient. So, I don't think we need to address it in
this KIP. Does that make sense?

Ismael

On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io> wrote:

> Ismael,
>           Sure AdminClient can do that and we should've shipped a config or
> use the existing one to block that. Not all users are yet to upgrade to
> AdminClient and start using that to cause  issues yet.
>              In shared environment we should allow server to set sane
> defaults and not allow every client to go ahead create random no.of
> topic/partitions and replication factor.   Even if the users want to allow
> topic creation proposed in the KIP , it makes sense to have some guards
> against the no.of partitions and replication factor. Authorizer is not
> always an answer to block requests and having users set server side configs
> to protect a multi-tenant environment is required.  In a non-secure
> environment Authorizer is a blunt instrument either you end up blocking
> everyone or allowing everyone.
> I am asking to have server side that allow clients to create topics or not
> , if they are allowed  set a ceiling on max no.of partitions and
> replication-factor.
>
> -Harsha
>
>
>
> On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:
>
> > Harsha,
> >
> > Rogue clients can use the admin client to create topics and partitions.
> > ACLs and policies can help in that case as well as this one.
> >
> > Ismael
> >
> > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io>
> wrote:
> >
> > Hi Justine,
> > Thanks for the KIP.
> > "When server-side auto-creation is disabled, client-side auto-creation
> > will try to use client-side configurations"
> > If I understand correctly, this KIP is removing any server-side blocking
> > client auto creation of topic?
> > if so this will present potential issue of rogue client creating ton of
> > topic-partitions and potentially bringing down the service for everyone
> or
> > degrade the service itself.
> > By reading the KIP its not clear to me that there is a clear way to block
> > auto creation topics of all together from clients by server side config.
> > Server side configs of default topic, partitions should take higher
> > precedence and client shouldn't be able to create a topic with higher
> no.of
> > partitions, replication than what server config specifies.
> >
> > >
> >
> > Thanks,
> > Harsha
> >
> > >
> > >
> > >
> >
> > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> > wrote:
> >
> > >
> >
> > > Hi all,
> > > I made some changes to the KIP. Hopefully this configuration change
> will
> > > make things a little clearer.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > >
> > > Please let me know if you have any feedback or questions!
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think you bring up a good point. It would be better if we didn't ever
> > > have to set up client-side configuration for this feature, and KIP-464
> > > would let us skip this entirely.
> > >
> > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> > >
> > > Hi Mickael,
> > > I agree that KIP-464 works on newer brokers, but I was a bit worried
> how
> > > things would play out on older brokers that* do not *have KIP 464
> > >
> > > included.
> > >
> > > Is it enough to throw an error in this case when producer configs are
> > not
> > > specified?
> > >
> > > I think the right thing to do would be to log an error message in the
> > > client. We will need to have that capability in any case, to cover
> > > scenarios like the client trying to auto-create a topic that they don't
> > > have permission to create. Or a client trying to create a topic on a
> > broker
> > > so old that CreateTopicsRequest is not supported.
> > >
> > > The big downside to relying on KIP-464 is that it is a very recent
> > feature
> > > -- so recent that it hasn't even made its way to any official Apache
> > > release. It's scheduled for the upcoming 2.4 release in a few months.
> > >
> > > So if you view this KIP as a step towards removing broker-side
> > > auto-create, you might want to support older brokers just to accelerate
> > > adoption, and hasten the day when we can finally flip broker-side
> > > auto-create to off (or even remove it entirely).
> > >
> > > I have to agree, though, that having client-side configurations for
> > number
> > > of partitions and replication factor is messy. Maybe it would be worth
> > it
> > > to restrict support to post-KIP-464 brokers, if we could avoid creating
> > > more configs.
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <
> mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > We can rely on KIP-464 which allows to omit the partition count or
> > > replication factor when creating a topic. In that case, the broker
> > defaults
> > > are used.
> > >
> > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > > wrote:
> > >
> > > Michael,
> > > That makes sense to me!
> > > To clarify, in the current state of the KIP, the producer does not
> > >
> > > rely
> > >
> > > on
> > >
> > > the broker to autocreate--if the broker's config is disabled, then
> > >
> > > the
> > >
> > > producer can autocreate on its own with a create topic request (the
> > >
> > > same
> > >
> > > type of request the admin client uses).
> > > However, if both configs are enabled, the broker will autocreate
> > >
> > > through
> > >
> > > a
> > >
> > > metadata request before the producer gets a chance. Of course, the way
> > to
> > > avoid this, is to do as you suggested, and set
> > >
> > > the
> > >
> > > "allow_auto_topic_creation" field to false.
> > >
> > > I think the only thing we need to be careful with in this setup is
> > >
> > > without
> > >
> > > KIP 464, we can not use broker defaults for this topic. A user needs
> > >
> > > to
> > >
> > > specify the number of partition and replication factor in the config.
> An
> > > alternative to this is to have coded defaults for when these
> > >
> > > configs
> > >
> > > are
> > >
> > > unspecified, but it is not immediately apparent what these defaults
> > >
> > > should
> > >
> > > be.
> > >
> > > Thanks again for reading my KIP,
> > > Justine
> > >
> > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the response!
> > > In my opinion, it would be better if the producer did not rely at
> > >
> > > all
> > >
> > > on the broker auto create feature as this is what we're aiming to
> > > deprecate. When requesting metadata we can set the
> > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > creation. Then if the topic is not existing, send a CreateTopicRequest.
> > >
> > > What do you think?
> > >
> > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Currently the way it is implemented, the broker auto-creation
> > >
> > > configuration
> > >
> > > takes precedence. The producer will not use the CreateTopics
> > >
> > > request.
> > >
> > > (Technically it can--but the topic will already be created
> > >
> > > through
> > >
> > > the
> > >
> > > broker, so it will never try to create the topic.) It is possible to
> > > change this however, and I'd be happy to
> > >
> > > discuss
> > >
> > > the
> > >
> > > benefits of this alternative.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP!
> > >
> > > In case auto creation is enabled on both the client and server,
> > >
> > > will
> > >
> > > the producer still use the AdminClient (CreateTopics request)
> > >
> > > to
> > >
> > > create topics? and not rely on the broker auto create. I'm guessing the
> > > answer is yes but can you make it explicit.
> > >
> > > Thank you
> > >
> > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi,
> > > Just a friendly reminder to take a look at this KIP if you
> > >
> > > have
> > >
> > > the
> > >
> > > time.
> > >
> > > I was thinking about broker vs. client default precedence,
> > >
> > > and I
> > >
> > > think it
> > >
> > > makes sense to keep the broker as the default used when both
> > >
> > > client-side
> > >
> > > and broker-side defaults are configured. The idea is that
> > >
> > > there
> > >
> > > would be
> > >
> > > pretty clear documentation, and that many systems with
> > >
> > > configurations
> > >
> > > that
> > >
> > > the client could not change would likely have the auto-create
> > >
> > > default
> > >
> > > off.
> > >
> > > (In cloud for example).
> > >
> > > It also seems like in most cases, the consumer config
> > > 'allow.auto.create.topics' was created to actually prevent
> > >
> > > the
> > >
> > > creation
> > >
> > > of
> > >
> > > topics, so the loss of creation functionality will not be a
> > >
> > > big
> > >
> > > problem.
> > >
> > > I'm happy to discuss any other compatibility problems or
> > >
> > > components
> > >
> > > of
> > >
> > > this KIP.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> > >
> > > jolshan@confluent.io
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I was looking at this KIP again, and there is a decision I
> > >
> > > made
> > >
> > > that I
> > >
> > > think is worth discussing.
> > >
> > > In the case where both the broker and producer's
> > > 'auto.create.topics.enable' are set to true, we have to
> > >
> > > choose
> > >
> > > either
> > >
> > > the
> > >
> > > broker configs or the producer configs for the replication
> > > factor/partitions.
> > >
> > > Currently, the decision is to have the broker defaults take
> > >
> > > precedence.
> > >
> > > (It is easier to do this in the implementation.) It also
> > >
> > > makes
> > >
> > > some
> > >
> > > sense
> > >
> > > for this behavior to take precedence since this behavior
> > >
> > > already
> > >
> > > occurs as
> > >
> > > the default.
> > >
> > > However, I was wondering if it would be odd for those who
> > >
> > > can
> > >
> > > only
> > >
> > > see
> > >
> > > the
> > >
> > > producer side to set configs for replication
> > >
> > > factor/partitions
> > >
> > > and
> > >
> > > see
> > >
> > > different results. Currently the documentation for the
> > >
> > > config
> > >
> > > states
> > >
> > > that
> > >
> > > the config values are only used when the broker config is
> > >
> > > not
> > >
> > > enabled,
> > >
> > > but
> > >
> > > this might not always be clear to the user. Changing the
> > >
> > > code
> > >
> > > to
> > >
> > > have
> > >
> > > the
> > >
> > > producer's configurations take precedence is possible, but
> > >
> > > I
> > >
> > > was
> > >
> > > wondering
> > >
> > > what everyone thought.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Just a quick update--
> > >
> > > It seems that enabling both the broker and producer
> > >
> > > configs
> > >
> > > works
> > >
> > > fine,
> > >
> > > except that the broker configurations for partitions,
> > >
> > > replication
> > >
> > > factor
> > >
> > > take precedence.
> > > I don't know if that is something we would want to
> > >
> > > change, but
> > >
> > > I'll be
> > >
> > > updating the KIP for now to reflect this. Perhaps we would
> > >
> > > want to
> > >
> > > add more
> > >
> > > to the documentation of the the producer configs to
> > >
> > > clarify.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for looking at the KIP. I can definitely add to
> > >
> > > the
> > >
> > > title
> > >
> > > to
> > >
> > > make
> > >
> > > it more clear.
> > >
> > > It makes sense that both configurations could be turned
> > >
> > > on
> > >
> > > since
> > >
> > > there
> > >
> > > are many cases where the user can not control the
> > >
> > > server-side
> > >
> > > configurations. I was a little concerned about how both
> > >
> > > interacting
> > >
> > > would
> > >
> > > work out -- if there would be to many requests for new
> > >
> > > topics,
> > >
> > > for
> > >
> > > example.
> > >
> > > But it since it does make sense to allow both
> > >
> > > configurations
> > >
> > > enabled, I
> > >
> > > will test out some scenarios and I'll change the KIP to
> > >
> > > support
> > >
> > > this.
> > >
> > > I also agree with documentation about distinguishing the
> > >
> > > differences. I
> > >
> > > was having some trouble with the wording but I like the
> > >
> > > phrases
> > >
> > > "server-side" and "client-side." That's a good
> > >
> > > distinction I
> > >
> > > can
> > >
> > > use
> > >
> > > when
> > >
> > > describing.
> > >
> > > I'll try to update the KIP soon keeping everyone's input
> > >
> > > in
> > >
> > > mind.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> > >
> > > cmccabe@apache.org
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP. This seems like a good step towards
> > >
> > > removing
> > >
> > > server-side topic auto-creation.
> > >
> > > We should add included "client-side" to the title of
> > >
> > > the KIP
> > >
> > > somewhere,
> > >
> > > to make it clear that we're talking about client-side
> > >
> > > auto
> > >
> > > creation.
> > >
> > > The KIP says:
> > >
> > > In order to automatically create topics with the
> > >
> > > producer, the
> > >
> > > producer's
> > >
> > > auto.create.topics.enable config must be set to true
> > >
> > > and
> > >
> > > the
> > >
> > > broker
> > >
> > > config should be set to false
> > >
> > > From a user's point of view, this seems
> > >
> > > counter-intuitive.
> > >
> > > In
> > >
> > > order to
> > >
> > > auto-create topics the broker's
> > >
> > > auto.create.topics.enable
> > >
> > > config
> > >
> > > should be
> > >
> > > set to false? It seems like the server-side
> > >
> > > auto-create is
> > >
> > > unrelated to
> > >
> > > the client-side auto-create. We could have both turned
> > >
> > > on
> > >
> > > (and
> > >
> > > I'm
> > >
> > > sure
> > >
> > > that in the real world, people will try this
> > >
> > > configuration...)
> > >
> > > There's no
> > >
> > > reason not to support this, I think.
> > >
> > > We should add some documentation explaining the
> > >
> > > difference
> > >
> > > between
> > >
> > > server-side and client-side auto-creation. Without
> > >
> > > documentation,
> > >
> > > an admin
> > >
> > > might think that they had disabled all forms of
> > >
> > > auto-creation by
> > >
> > > setting
> > >
> > > the -side setting to false-- but this is not the case,
> > >
> > > of
> > >
> > > course.
> > >
> > > best,
> > > Colin
> > >
> > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> > >
> > > Hi Dhruvil,
> > >
> > > Thanks for reading the KIP!
> > > That was the general idea for deprecation. We would
> > >
> > > log a
> > >
> > > warning
> > >
> > > when the
> > >
> > > config is enabled on the broker.
> > > I also believe that there would be a change to
> > >
> > > documentation.
> > >
> > > If there is anything else that should be done, please
> > >
> > > let
> > >
> > > me
> > >
> > > know!
> > >
> > > Justine
> > >
> > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> > >
> > > dhruvil@confluent.io>
> > >
> > > wrote:
> > >
> > > Hi Justine,
> > >
> > > Thanks for the KIP, this is great!
> > >
> > > Could you add some more information about what
> > >
> > > deprecating
> > >
> > > the
> > >
> > > broker
> > >
> > > configuration means? Would we log a warning in the
> > >
> > > logs
> > >
> > > when
> > >
> > > auto
> > >
> > > topic
> > >
> > > creation is enabled on the broker, for example?
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> > >
> > > jolshan@confluent.io>
> > >
> > > wrote:
> > >
> > > Hello all,
> > >
> > > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > > deprecate the current system of
> > >
> > > auto-creating
> > >
> > > topics
> > >
> > > through requests to the metadata and give the
> > >
> > > producer the
> > >
> > > ability to
> > >
> > > automatically create topics instead.
> > >
> > > More information can be found here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> > >
> > > Thank you,
> > > Justine Olshan
> > >
> > >
> >
> > >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Harsha Chintalapani <ka...@harsha.io>.

Ismael,
          Sure AdminClient can do that and we should've shipped a config or
use the existing one to block that. Not all users are yet to upgrade to
AdminClient and start using that to cause  issues yet.
             In shared environment we should allow server to set sane
defaults and not allow every client to go ahead create random no.of
topic/partitions and replication factor.   Even if the users want to allow
topic creation proposed in the KIP , it makes sense to have some guards
against the no.of partitions and replication factor. Authorizer is not
always an answer to block requests and having users set server side configs
to protect a multi-tenant environment is required.  In a non-secure
environment Authorizer is a blunt instrument either you end up blocking
everyone or allowing everyone.
I am asking to have server side that allow clients to create topics or not
, if they are allowed  set a ceiling on max no.of partitions and
replication-factor.

-Harsha



On Mon, Aug 5 2019 at 8:58 PM, <is...@gmail.com> wrote:

> Harsha,
>
> Rogue clients can use the admin client to create topics and partitions.
> ACLs and policies can help in that case as well as this one.
>
> Ismael
>
> On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io> wrote:
>
> Hi Justine,
> Thanks for the KIP.
> "When server-side auto-creation is disabled, client-side auto-creation
> will try to use client-side configurations"
> If I understand correctly, this KIP is removing any server-side blocking
> client auto creation of topic?
> if so this will present potential issue of rogue client creating ton of
> topic-partitions and potentially bringing down the service for everyone or
> degrade the service itself.
> By reading the KIP its not clear to me that there is a clear way to block
> auto creation topics of all together from clients by server side config.
> Server side configs of default topic, partitions should take higher
> precedence and client shouldn't be able to create a topic with higher no.of
> partitions, replication than what server config specifies.
>
> >
>
> Thanks,
> Harsha
>
> >
> >
> >
>
> On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> wrote:
>
> >
>
> > Hi all,
> > I made some changes to the KIP. Hopefully this configuration change will
> > make things a little clearer.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> >
> > Please let me know if you have any feedback or questions!
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org>
> wrote:
> >
> > Hi Mickael,
> >
> > I think you bring up a good point. It would be better if we didn't ever
> > have to set up client-side configuration for this feature, and KIP-464
> > would let us skip this entirely.
> >
> > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> >
> > Hi Mickael,
> > I agree that KIP-464 works on newer brokers, but I was a bit worried how
> > things would play out on older brokers that* do not *have KIP 464
> >
> > included.
> >
> > Is it enough to throw an error in this case when producer configs are
> not
> > specified?
> >
> > I think the right thing to do would be to log an error message in the
> > client. We will need to have that capability in any case, to cover
> > scenarios like the client trying to auto-create a topic that they don't
> > have permission to create. Or a client trying to create a topic on a
> broker
> > so old that CreateTopicsRequest is not supported.
> >
> > The big downside to relying on KIP-464 is that it is a very recent
> feature
> > -- so recent that it hasn't even made its way to any official Apache
> > release. It's scheduled for the upcoming 2.4 release in a few months.
> >
> > So if you view this KIP as a step towards removing broker-side
> > auto-create, you might want to support older brokers just to accelerate
> > adoption, and hasten the day when we can finally flip broker-side
> > auto-create to off (or even remove it entirely).
> >
> > I have to agree, though, that having client-side configurations for
> number
> > of partitions and replication factor is messy. Maybe it would be worth
> it
> > to restrict support to post-KIP-464 brokers, if we could avoid creating
> > more configs.
> >
> > best,
> > Colin
> >
> > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <mickael.maison@gmail.com
> >
> > wrote:
> >
> > Hi Justine,
> >
> > We can rely on KIP-464 which allows to omit the partition count or
> > replication factor when creating a topic. In that case, the broker
> defaults
> > are used.
> >
> > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > wrote:
> >
> > Michael,
> > That makes sense to me!
> > To clarify, in the current state of the KIP, the producer does not
> >
> > rely
> >
> > on
> >
> > the broker to autocreate--if the broker's config is disabled, then
> >
> > the
> >
> > producer can autocreate on its own with a create topic request (the
> >
> > same
> >
> > type of request the admin client uses).
> > However, if both configs are enabled, the broker will autocreate
> >
> > through
> >
> > a
> >
> > metadata request before the producer gets a chance. Of course, the way
> to
> > avoid this, is to do as you suggested, and set
> >
> > the
> >
> > "allow_auto_topic_creation" field to false.
> >
> > I think the only thing we need to be careful with in this setup is
> >
> > without
> >
> > KIP 464, we can not use broker defaults for this topic. A user needs
> >
> > to
> >
> > specify the number of partition and replication factor in the config. An
> > alternative to this is to have coded defaults for when these
> >
> > configs
> >
> > are
> >
> > unspecified, but it is not immediately apparent what these defaults
> >
> > should
> >
> > be.
> >
> > Thanks again for reading my KIP,
> > Justine
> >
> > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> >
> > mickael.maison@gmail.com
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the response!
> > In my opinion, it would be better if the producer did not rely at
> >
> > all
> >
> > on the broker auto create feature as this is what we're aiming to
> > deprecate. When requesting metadata we can set the
> > "allow_auto_topic_creation" field to false to avoid the broker auto
> > creation. Then if the topic is not existing, send a CreateTopicRequest.
> >
> > What do you think?
> >
> > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Currently the way it is implemented, the broker auto-creation
> >
> > configuration
> >
> > takes precedence. The producer will not use the CreateTopics
> >
> > request.
> >
> > (Technically it can--but the topic will already be created
> >
> > through
> >
> > the
> >
> > broker, so it will never try to create the topic.) It is possible to
> > change this however, and I'd be happy to
> >
> > discuss
> >
> > the
> >
> > benefits of this alternative.
> >
> > Thank you,
> > Justine
> >
> > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> >
> > mickael.maison@gmail.com>
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP!
> >
> > In case auto creation is enabled on both the client and server,
> >
> > will
> >
> > the producer still use the AdminClient (CreateTopics request)
> >
> > to
> >
> > create topics? and not rely on the broker auto create. I'm guessing the
> > answer is yes but can you make it explicit.
> >
> > Thank you
> >
> > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hi,
> > Just a friendly reminder to take a look at this KIP if you
> >
> > have
> >
> > the
> >
> > time.
> >
> > I was thinking about broker vs. client default precedence,
> >
> > and I
> >
> > think it
> >
> > makes sense to keep the broker as the default used when both
> >
> > client-side
> >
> > and broker-side defaults are configured. The idea is that
> >
> > there
> >
> > would be
> >
> > pretty clear documentation, and that many systems with
> >
> > configurations
> >
> > that
> >
> > the client could not change would likely have the auto-create
> >
> > default
> >
> > off.
> >
> > (In cloud for example).
> >
> > It also seems like in most cases, the consumer config
> > 'allow.auto.create.topics' was created to actually prevent
> >
> > the
> >
> > creation
> >
> > of
> >
> > topics, so the loss of creation functionality will not be a
> >
> > big
> >
> > problem.
> >
> > I'm happy to discuss any other compatibility problems or
> >
> > components
> >
> > of
> >
> > this KIP.
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> >
> > jolshan@confluent.io
> >
> > wrote:
> >
> > Hello all,
> >
> > I was looking at this KIP again, and there is a decision I
> >
> > made
> >
> > that I
> >
> > think is worth discussing.
> >
> > In the case where both the broker and producer's
> > 'auto.create.topics.enable' are set to true, we have to
> >
> > choose
> >
> > either
> >
> > the
> >
> > broker configs or the producer configs for the replication
> > factor/partitions.
> >
> > Currently, the decision is to have the broker defaults take
> >
> > precedence.
> >
> > (It is easier to do this in the implementation.) It also
> >
> > makes
> >
> > some
> >
> > sense
> >
> > for this behavior to take precedence since this behavior
> >
> > already
> >
> > occurs as
> >
> > the default.
> >
> > However, I was wondering if it would be odd for those who
> >
> > can
> >
> > only
> >
> > see
> >
> > the
> >
> > producer side to set configs for replication
> >
> > factor/partitions
> >
> > and
> >
> > see
> >
> > different results. Currently the documentation for the
> >
> > config
> >
> > states
> >
> > that
> >
> > the config values are only used when the broker config is
> >
> > not
> >
> > enabled,
> >
> > but
> >
> > this might not always be clear to the user. Changing the
> >
> > code
> >
> > to
> >
> > have
> >
> > the
> >
> > producer's configurations take precedence is possible, but
> >
> > I
> >
> > was
> >
> > wondering
> >
> > what everyone thought.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Just a quick update--
> >
> > It seems that enabling both the broker and producer
> >
> > configs
> >
> > works
> >
> > fine,
> >
> > except that the broker configurations for partitions,
> >
> > replication
> >
> > factor
> >
> > take precedence.
> > I don't know if that is something we would want to
> >
> > change, but
> >
> > I'll be
> >
> > updating the KIP for now to reflect this. Perhaps we would
> >
> > want to
> >
> > add more
> >
> > to the documentation of the the producer configs to
> >
> > clarify.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hi Colin,
> >
> > Thanks for looking at the KIP. I can definitely add to
> >
> > the
> >
> > title
> >
> > to
> >
> > make
> >
> > it more clear.
> >
> > It makes sense that both configurations could be turned
> >
> > on
> >
> > since
> >
> > there
> >
> > are many cases where the user can not control the
> >
> > server-side
> >
> > configurations. I was a little concerned about how both
> >
> > interacting
> >
> > would
> >
> > work out -- if there would be to many requests for new
> >
> > topics,
> >
> > for
> >
> > example.
> >
> > But it since it does make sense to allow both
> >
> > configurations
> >
> > enabled, I
> >
> > will test out some scenarios and I'll change the KIP to
> >
> > support
> >
> > this.
> >
> > I also agree with documentation about distinguishing the
> >
> > differences. I
> >
> > was having some trouble with the wording but I like the
> >
> > phrases
> >
> > "server-side" and "client-side." That's a good
> >
> > distinction I
> >
> > can
> >
> > use
> >
> > when
> >
> > describing.
> >
> > I'll try to update the KIP soon keeping everyone's input
> >
> > in
> >
> > mind.
> >
> > Thanks,
> > Justine
> >
> > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> >
> > cmccabe@apache.org
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP. This seems like a good step towards
> >
> > removing
> >
> > server-side topic auto-creation.
> >
> > We should add included "client-side" to the title of
> >
> > the KIP
> >
> > somewhere,
> >
> > to make it clear that we're talking about client-side
> >
> > auto
> >
> > creation.
> >
> > The KIP says:
> >
> > In order to automatically create topics with the
> >
> > producer, the
> >
> > producer's
> >
> > auto.create.topics.enable config must be set to true
> >
> > and
> >
> > the
> >
> > broker
> >
> > config should be set to false
> >
> > From a user's point of view, this seems
> >
> > counter-intuitive.
> >
> > In
> >
> > order to
> >
> > auto-create topics the broker's
> >
> > auto.create.topics.enable
> >
> > config
> >
> > should be
> >
> > set to false? It seems like the server-side
> >
> > auto-create is
> >
> > unrelated to
> >
> > the client-side auto-create. We could have both turned
> >
> > on
> >
> > (and
> >
> > I'm
> >
> > sure
> >
> > that in the real world, people will try this
> >
> > configuration...)
> >
> > There's no
> >
> > reason not to support this, I think.
> >
> > We should add some documentation explaining the
> >
> > difference
> >
> > between
> >
> > server-side and client-side auto-creation. Without
> >
> > documentation,
> >
> > an admin
> >
> > might think that they had disabled all forms of
> >
> > auto-creation by
> >
> > setting
> >
> > the -side setting to false-- but this is not the case,
> >
> > of
> >
> > course.
> >
> > best,
> > Colin
> >
> > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> >
> > Hi Dhruvil,
> >
> > Thanks for reading the KIP!
> > That was the general idea for deprecation. We would
> >
> > log a
> >
> > warning
> >
> > when the
> >
> > config is enabled on the broker.
> > I also believe that there would be a change to
> >
> > documentation.
> >
> > If there is anything else that should be done, please
> >
> > let
> >
> > me
> >
> > know!
> >
> > Justine
> >
> > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> >
> > dhruvil@confluent.io>
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP, this is great!
> >
> > Could you add some more information about what
> >
> > deprecating
> >
> > the
> >
> > broker
> >
> > configuration means? Would we log a warning in the
> >
> > logs
> >
> > when
> >
> > auto
> >
> > topic
> >
> > creation is enabled on the broker, for example?
> >
> > Thanks,
> > Dhruvil
> >
> > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hello all,
> >
> > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > deprecate the current system of
> >
> > auto-creating
> >
> > topics
> >
> > through requests to the metadata and give the
> >
> > producer the
> >
> > ability to
> >
> > automatically create topics instead.
> >
> > More information can be found here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> >
> > Thank you,
> > Justine Olshan
> >
> >
>
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Ismael Juma <is...@gmail.com>.

Harsha,

Rogue clients can use the admin client to create topics and partitions.
ACLs and policies can help in that case as well as this one.

Ismael

On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io> wrote:

> Hi Justine,
>              Thanks for the KIP.
> "When server-side auto-creation is disabled, client-side auto-creation will
> try to use client-side configurations"
> If I understand correctly, this KIP is removing any server-side blocking
> client auto creation of topic?
> if so this will present potential issue of rogue client creating ton of
> topic-partitions and potentially bringing down the service for everyone or
> degrade the service itself.
> By reading the KIP its not clear to me that there is a clear way to block
> auto creation topics of all together from clients by server side config.
> Server side configs of default topic, partitions should take higher
> precedence and client shouldn't be able to create a topic with higher no.of
> partitions, replication than what server config specifies.
>
> Thanks,
> Harsha
>
>
>
> On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
> wrote:
>
> > Hi all,
> > I made some changes to the KIP. Hopefully this configuration change will
> > make things a little clearer.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> >
> > Please let me know if you have any feedback or questions!
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > Hi Mickael,
> >
> > I think you bring up a good point. It would be better if we didn't ever
> > have to set up client-side configuration for this feature, and KIP-464
> > would let us skip this entirely.
> >
> > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> >
> > Hi Mickael,
> > I agree that KIP-464 works on newer brokers, but I was a bit worried how
> > things would play out on older brokers that* do not *have KIP 464
> >
> > included.
> >
> > Is it enough to throw an error in this case when producer configs are not
> > specified?
> >
> > I think the right thing to do would be to log an error message in the
> > client. We will need to have that capability in any case, to cover
> > scenarios like the client trying to auto-create a topic that they don't
> > have permission to create. Or a client trying to create a topic on a
> broker
> > so old that CreateTopicsRequest is not supported.
> >
> > The big downside to relying on KIP-464 is that it is a very recent
> feature
> > -- so recent that it hasn't even made its way to any official Apache
> > release. It's scheduled for the upcoming 2.4 release in a few months.
> >
> > So if you view this KIP as a step towards removing broker-side
> > auto-create, you might want to support older brokers just to accelerate
> > adoption, and hasten the day when we can finally flip broker-side
> > auto-create to off (or even remove it entirely).
> >
> > I have to agree, though, that having client-side configurations for
> number
> > of partitions and replication factor is messy. Maybe it would be worth it
> > to restrict support to post-KIP-464 brokers, if we could avoid creating
> > more configs.
> >
> > best,
> > Colin
> >
> > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <mickael.maison@gmail.com
> >
> > wrote:
> >
> > Hi Justine,
> >
> > We can rely on KIP-464 which allows to omit the partition count or
> > replication factor when creating a topic. In that case, the broker
> defaults
> > are used.
> >
> > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> > wrote:
> >
> > Michael,
> > That makes sense to me!
> > To clarify, in the current state of the KIP, the producer does not
> >
> > rely
> >
> > on
> >
> > the broker to autocreate--if the broker's config is disabled, then
> >
> > the
> >
> > producer can autocreate on its own with a create topic request (the
> >
> > same
> >
> > type of request the admin client uses).
> > However, if both configs are enabled, the broker will autocreate
> >
> > through
> >
> > a
> >
> > metadata request before the producer gets a chance. Of course, the way to
> > avoid this, is to do as you suggested, and set
> >
> > the
> >
> > "allow_auto_topic_creation" field to false.
> >
> > I think the only thing we need to be careful with in this setup is
> >
> > without
> >
> > KIP 464, we can not use broker defaults for this topic. A user needs
> >
> > to
> >
> > specify the number of partition and replication factor in the config. An
> > alternative to this is to have coded defaults for when these
> >
> > configs
> >
> > are
> >
> > unspecified, but it is not immediately apparent what these defaults
> >
> > should
> >
> > be.
> >
> > Thanks again for reading my KIP,
> > Justine
> >
> > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
> >
> > mickael.maison@gmail.com
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the response!
> > In my opinion, it would be better if the producer did not rely at
> >
> > all
> >
> > on the broker auto create feature as this is what we're aiming to
> > deprecate. When requesting metadata we can set the
> > "allow_auto_topic_creation" field to false to avoid the broker auto
> > creation. Then if the topic is not existing, send a CreateTopicRequest.
> >
> > What do you think?
> >
> > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Currently the way it is implemented, the broker auto-creation
> >
> > configuration
> >
> > takes precedence. The producer will not use the CreateTopics
> >
> > request.
> >
> > (Technically it can--but the topic will already be created
> >
> > through
> >
> > the
> >
> > broker, so it will never try to create the topic.) It is possible to
> > change this however, and I'd be happy to
> >
> > discuss
> >
> > the
> >
> > benefits of this alternative.
> >
> > Thank you,
> > Justine
> >
> > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> >
> > mickael.maison@gmail.com>
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP!
> >
> > In case auto creation is enabled on both the client and server,
> >
> > will
> >
> > the producer still use the AdminClient (CreateTopics request)
> >
> > to
> >
> > create topics? and not rely on the broker auto create. I'm guessing the
> > answer is yes but can you make it explicit.
> >
> > Thank you
> >
> > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hi,
> > Just a friendly reminder to take a look at this KIP if you
> >
> > have
> >
> > the
> >
> > time.
> >
> > I was thinking about broker vs. client default precedence,
> >
> > and I
> >
> > think it
> >
> > makes sense to keep the broker as the default used when both
> >
> > client-side
> >
> > and broker-side defaults are configured. The idea is that
> >
> > there
> >
> > would be
> >
> > pretty clear documentation, and that many systems with
> >
> > configurations
> >
> > that
> >
> > the client could not change would likely have the auto-create
> >
> > default
> >
> > off.
> >
> > (In cloud for example).
> >
> > It also seems like in most cases, the consumer config
> > 'allow.auto.create.topics' was created to actually prevent
> >
> > the
> >
> > creation
> >
> > of
> >
> > topics, so the loss of creation functionality will not be a
> >
> > big
> >
> > problem.
> >
> > I'm happy to discuss any other compatibility problems or
> >
> > components
> >
> > of
> >
> > this KIP.
> >
> > Thank you,
> > Justine
> >
> > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> >
> > jolshan@confluent.io
> >
> > wrote:
> >
> > Hello all,
> >
> > I was looking at this KIP again, and there is a decision I
> >
> > made
> >
> > that I
> >
> > think is worth discussing.
> >
> > In the case where both the broker and producer's
> > 'auto.create.topics.enable' are set to true, we have to
> >
> > choose
> >
> > either
> >
> > the
> >
> > broker configs or the producer configs for the replication
> > factor/partitions.
> >
> > Currently, the decision is to have the broker defaults take
> >
> > precedence.
> >
> > (It is easier to do this in the implementation.) It also
> >
> > makes
> >
> > some
> >
> > sense
> >
> > for this behavior to take precedence since this behavior
> >
> > already
> >
> > occurs as
> >
> > the default.
> >
> > However, I was wondering if it would be odd for those who
> >
> > can
> >
> > only
> >
> > see
> >
> > the
> >
> > producer side to set configs for replication
> >
> > factor/partitions
> >
> > and
> >
> > see
> >
> > different results. Currently the documentation for the
> >
> > config
> >
> > states
> >
> > that
> >
> > the config values are only used when the broker config is
> >
> > not
> >
> > enabled,
> >
> > but
> >
> > this might not always be clear to the user. Changing the
> >
> > code
> >
> > to
> >
> > have
> >
> > the
> >
> > producer's configurations take precedence is possible, but
> >
> > I
> >
> > was
> >
> > wondering
> >
> > what everyone thought.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Just a quick update--
> >
> > It seems that enabling both the broker and producer
> >
> > configs
> >
> > works
> >
> > fine,
> >
> > except that the broker configurations for partitions,
> >
> > replication
> >
> > factor
> >
> > take precedence.
> > I don't know if that is something we would want to
> >
> > change, but
> >
> > I'll be
> >
> > updating the KIP for now to reflect this. Perhaps we would
> >
> > want to
> >
> > add more
> >
> > to the documentation of the the producer configs to
> >
> > clarify.
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hi Colin,
> >
> > Thanks for looking at the KIP. I can definitely add to
> >
> > the
> >
> > title
> >
> > to
> >
> > make
> >
> > it more clear.
> >
> > It makes sense that both configurations could be turned
> >
> > on
> >
> > since
> >
> > there
> >
> > are many cases where the user can not control the
> >
> > server-side
> >
> > configurations. I was a little concerned about how both
> >
> > interacting
> >
> > would
> >
> > work out -- if there would be to many requests for new
> >
> > topics,
> >
> > for
> >
> > example.
> >
> > But it since it does make sense to allow both
> >
> > configurations
> >
> > enabled, I
> >
> > will test out some scenarios and I'll change the KIP to
> >
> > support
> >
> > this.
> >
> > I also agree with documentation about distinguishing the
> >
> > differences. I
> >
> > was having some trouble with the wording but I like the
> >
> > phrases
> >
> > "server-side" and "client-side." That's a good
> >
> > distinction I
> >
> > can
> >
> > use
> >
> > when
> >
> > describing.
> >
> > I'll try to update the KIP soon keeping everyone's input
> >
> > in
> >
> > mind.
> >
> > Thanks,
> > Justine
> >
> > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
> >
> > cmccabe@apache.org
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP. This seems like a good step towards
> >
> > removing
> >
> > server-side topic auto-creation.
> >
> > We should add included "client-side" to the title of
> >
> > the KIP
> >
> > somewhere,
> >
> > to make it clear that we're talking about client-side
> >
> > auto
> >
> > creation.
> >
> > The KIP says:
> >
> > In order to automatically create topics with the
> >
> > producer, the
> >
> > producer's
> >
> > auto.create.topics.enable config must be set to true
> >
> > and
> >
> > the
> >
> > broker
> >
> > config should be set to false
> >
> > From a user's point of view, this seems
> >
> > counter-intuitive.
> >
> > In
> >
> > order to
> >
> > auto-create topics the broker's
> >
> > auto.create.topics.enable
> >
> > config
> >
> > should be
> >
> > set to false? It seems like the server-side
> >
> > auto-create is
> >
> > unrelated to
> >
> > the client-side auto-create. We could have both turned
> >
> > on
> >
> > (and
> >
> > I'm
> >
> > sure
> >
> > that in the real world, people will try this
> >
> > configuration...)
> >
> > There's no
> >
> > reason not to support this, I think.
> >
> > We should add some documentation explaining the
> >
> > difference
> >
> > between
> >
> > server-side and client-side auto-creation. Without
> >
> > documentation,
> >
> > an admin
> >
> > might think that they had disabled all forms of
> >
> > auto-creation by
> >
> > setting
> >
> > the -side setting to false-- but this is not the case,
> >
> > of
> >
> > course.
> >
> > best,
> > Colin
> >
> > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
> >
> > Hi Dhruvil,
> >
> > Thanks for reading the KIP!
> > That was the general idea for deprecation. We would
> >
> > log a
> >
> > warning
> >
> > when the
> >
> > config is enabled on the broker.
> > I also believe that there would be a change to
> >
> > documentation.
> >
> > If there is anything else that should be done, please
> >
> > let
> >
> > me
> >
> > know!
> >
> > Justine
> >
> > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
> >
> > dhruvil@confluent.io>
> >
> > wrote:
> >
> > Hi Justine,
> >
> > Thanks for the KIP, this is great!
> >
> > Could you add some more information about what
> >
> > deprecating
> >
> > the
> >
> > broker
> >
> > configuration means? Would we log a warning in the
> >
> > logs
> >
> > when
> >
> > auto
> >
> > topic
> >
> > creation is enabled on the broker, for example?
> >
> > Thanks,
> > Dhruvil
> >
> > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
> >
> > jolshan@confluent.io>
> >
> > wrote:
> >
> > Hello all,
> >
> > I'd like to start a discussion thread for KIP-487. This KIP plans to
> > deprecate the current system of
> >
> > auto-creating
> >
> > topics
> >
> > through requests to the metadata and give the
> >
> > producer the
> >
> > ability to
> >
> > automatically create topics instead.
> >
> > More information can be found here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-487%3A+Automatic+Topic+Creation+on+Producer
> >
> > Thank you,
> > Justine Olshan
> >
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Posted by Harsha Chintalapani <ka...@harsha.io>.

Hi Justine,
             Thanks for the KIP.
"When server-side auto-creation is disabled, client-side auto-creation will
try to use client-side configurations"
If I understand correctly, this KIP is removing any server-side blocking
client auto creation of topic?
if so this will present potential issue of rogue client creating ton of
topic-partitions and potentially bringing down the service for everyone or
degrade the service itself.
By reading the KIP its not clear to me that there is a clear way to block
auto creation topics of all together from clients by server side config.
Server side configs of default topic, partitions should take higher
precedence and client shouldn't be able to create a topic with higher no.of
partitions, replication than what server config specifies.

Thanks,
Harsha



On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan <jo...@confluent.io>
wrote:

> Hi all,
> I made some changes to the KIP. Hopefully this configuration change will
> make things a little clearer.
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
>
> Please let me know if you have any feedback or questions!
>
> Thank you,
> Justine
>
> On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cm...@apache.org> wrote:
>
> Hi Mickael,
>
> I think you bring up a good point. It would be better if we didn't ever
> have to set up client-side configuration for this feature, and KIP-464
> would let us skip this entirely.
>
> On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
>
> Hi Mickael,
> I agree that KIP-464 works on newer brokers, but I was a bit worried how
> things would play out on older brokers that* do not *have KIP 464
>
> included.
>
> Is it enough to throw an error in this case when producer configs are not
> specified?
>
> I think the right thing to do would be to log an error message in the
> client. We will need to have that capability in any case, to cover
> scenarios like the client trying to auto-create a topic that they don't
> have permission to create. Or a client trying to create a topic on a broker
> so old that CreateTopicsRequest is not supported.
>
> The big downside to relying on KIP-464 is that it is a very recent feature
> -- so recent that it hasn't even made its way to any official Apache
> release. It's scheduled for the upcoming 2.4 release in a few months.
>
> So if you view this KIP as a step towards removing broker-side
> auto-create, you might want to support older brokers just to accelerate
> adoption, and hasten the day when we can finally flip broker-side
> auto-create to off (or even remove it entirely).
>
> I have to agree, though, that having client-side configurations for number
> of partitions and replication factor is messy. Maybe it would be worth it
> to restrict support to post-KIP-464 brokers, if we could avoid creating
> more configs.
>
> best,
> Colin
>
> On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <mickael.maison@gmail.com
>
> wrote:
>
> Hi Justine,
>
> We can rely on KIP-464 which allows to omit the partition count or
> replication factor when creating a topic. In that case, the broker defaults
> are used.
>
> On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jo...@confluent.io>
> wrote:
>
> Michael,
> That makes sense to me!
> To clarify, in the current state of the KIP, the producer does not
>
> rely
>
> on
>
> the broker to autocreate--if the broker's config is disabled, then
>
> the
>
> producer can autocreate on its own with a create topic request (the
>
> same
>
> type of request the admin client uses).
> However, if both configs are enabled, the broker will autocreate
>
> through
>
> a
>
> metadata request before the producer gets a chance. Of course, the way to
> avoid this, is to do as you suggested, and set
>
> the
>
> "allow_auto_topic_creation" field to false.
>
> I think the only thing we need to be careful with in this setup is
>
> without
>
> KIP 464, we can not use broker defaults for this topic. A user needs
>
> to
>
> specify the number of partition and replication factor in the config. An
> alternative to this is to have coded defaults for when these
>
> configs
>
> are
>
> unspecified, but it is not immediately apparent what these defaults
>
> should
>
> be.
>
> Thanks again for reading my KIP,
> Justine
>
> On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <
>
> mickael.maison@gmail.com
>
> wrote:
>
> Hi Justine,
>
> Thanks for the response!
> In my opinion, it would be better if the producer did not rely at
>
> all
>
> on the broker auto create feature as this is what we're aiming to
> deprecate. When requesting metadata we can set the
> "allow_auto_topic_creation" field to false to avoid the broker auto
> creation. Then if the topic is not existing, send a CreateTopicRequest.
>
> What do you think?
>
> On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Currently the way it is implemented, the broker auto-creation
>
> configuration
>
> takes precedence. The producer will not use the CreateTopics
>
> request.
>
> (Technically it can--but the topic will already be created
>
> through
>
> the
>
> broker, so it will never try to create the topic.) It is possible to
> change this however, and I'd be happy to
>
> discuss
>
> the
>
> benefits of this alternative.
>
> Thank you,
> Justine
>
> On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
>
> mickael.maison@gmail.com>
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP!
>
> In case auto creation is enabled on both the client and server,
>
> will
>
> the producer still use the AdminClient (CreateTopics request)
>
> to
>
> create topics? and not rely on the broker auto create. I'm guessing the
> answer is yes but can you make it explicit.
>
> Thank you
>
> On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Hi,
> Just a friendly reminder to take a look at this KIP if you
>
> have
>
> the
>
> time.
>
> I was thinking about broker vs. client default precedence,
>
> and I
>
> think it
>
> makes sense to keep the broker as the default used when both
>
> client-side
>
> and broker-side defaults are configured. The idea is that
>
> there
>
> would be
>
> pretty clear documentation, and that many systems with
>
> configurations
>
> that
>
> the client could not change would likely have the auto-create
>
> default
>
> off.
>
> (In cloud for example).
>
> It also seems like in most cases, the consumer config
> 'allow.auto.create.topics' was created to actually prevent
>
> the
>
> creation
>
> of
>
> topics, so the loss of creation functionality will not be a
>
> big
>
> problem.
>
> I'm happy to discuss any other compatibility problems or
>
> components
>
> of
>
> this KIP.
>
> Thank you,
> Justine
>
> On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
>
> jolshan@confluent.io
>
> wrote:
>
> Hello all,
>
> I was looking at this KIP again, and there is a decision I
>
> made
>
> that I
>
> think is worth discussing.
>
> In the case where both the broker and producer's
> 'auto.create.topics.enable' are set to true, we have to
>
> choose
>
> either
>
> the
>
> broker configs or the producer configs for the replication
> factor/partitions.
>
> Currently, the decision is to have the broker defaults take
>
> precedence.
>
> (It is easier to do this in the implementation.) It also
>
> makes
>
> some
>
> sense
>
> for this behavior to take precedence since this behavior
>
> already
>
> occurs as
>
> the default.
>
> However, I was wondering if it would be odd for those who
>
> can
>
> only
>
> see
>
> the
>
> producer side to set configs for replication
>
> factor/partitions
>
> and
>
> see
>
> different results. Currently the documentation for the
>
> config
>
> states
>
> that
>
> the config values are only used when the broker config is
>
> not
>
> enabled,
>
> but
>
> this might not always be clear to the user. Changing the
>
> code
>
> to
>
> have
>
> the
>
> producer's configurations take precedence is possible, but
>
> I
>
> was
>
> wondering
>
> what everyone thought.
>
> Thank you,
> Justine
>
> On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Just a quick update--
>
> It seems that enabling both the broker and producer
>
> configs
>
> works
>
> fine,
>
> except that the broker configurations for partitions,
>
> replication
>
> factor
>
> take precedence.
> I don't know if that is something we would want to
>
> change, but
>
> I'll be
>
> updating the KIP for now to reflect this. Perhaps we would
>
> want to
>
> add more
>
> to the documentation of the the producer configs to
>
> clarify.
>
> Thank you,
> Justine
>
> On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Hi Colin,
>
> Thanks for looking at the KIP. I can definitely add to
>
> the
>
> title
>
> to
>
> make
>
> it more clear.
>
> It makes sense that both configurations could be turned
>
> on
>
> since
>
> there
>
> are many cases where the user can not control the
>
> server-side
>
> configurations. I was a little concerned about how both
>
> interacting
>
> would
>
> work out -- if there would be to many requests for new
>
> topics,
>
> for
>
> example.
>
> But it since it does make sense to allow both
>
> configurations
>
> enabled, I
>
> will test out some scenarios and I'll change the KIP to
>
> support
>
> this.
>
> I also agree with documentation about distinguishing the
>
> differences. I
>
> was having some trouble with the wording but I like the
>
> phrases
>
> "server-side" and "client-side." That's a good
>
> distinction I
>
> can
>
> use
>
> when
>
> describing.
>
> I'll try to update the KIP soon keeping everyone's input
>
> in
>
> mind.
>
> Thanks,
> Justine
>
> On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <
>
> cmccabe@apache.org
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP. This seems like a good step towards
>
> removing
>
> server-side topic auto-creation.
>
> We should add included "client-side" to the title of
>
> the KIP
>
> somewhere,
>
> to make it clear that we're talking about client-side
>
> auto
>
> creation.
>
> The KIP says:
>
> In order to automatically create topics with the
>
> producer, the
>
> producer's
>
> auto.create.topics.enable config must be set to true
>
> and
>
> the
>
> broker
>
> config should be set to false
>
> From a user's point of view, this seems
>
> counter-intuitive.
>
> In
>
> order to
>
> auto-create topics the broker's
>
> auto.create.topics.enable
>
> config
>
> should be
>
> set to false? It seems like the server-side
>
> auto-create is
>
> unrelated to
>
> the client-side auto-create. We could have both turned
>
> on
>
> (and
>
> I'm
>
> sure
>
> that in the real world, people will try this
>
> configuration...)
>
> There's no
>
> reason not to support this, I think.
>
> We should add some documentation explaining the
>
> difference
>
> between
>
> server-side and client-side auto-creation. Without
>
> documentation,
>
> an admin
>
> might think that they had disabled all forms of
>
> auto-creation by
>
> setting
>
> the -side setting to false-- but this is not the case,
>
> of
>
> course.
>
> best,
> Colin
>
> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote:
>
> Hi Dhruvil,
>
> Thanks for reading the KIP!
> That was the general idea for deprecation. We would
>
> log a
>
> warning
>
> when the
>
> config is enabled on the broker.
> I also believe that there would be a change to
>
> documentation.
>
> If there is anything else that should be done, please
>
> let
>
> me
>
> know!
>
> Justine
>
> On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <
>
> dhruvil@confluent.io>
>
> wrote:
>
> Hi Justine,
>
> Thanks for the KIP, this is great!
>
> Could you add some more information about what
>
> deprecating
>
> the
>
> broker
>
> configuration means? Would we log a warning in the
>
> logs
>
> when
>
> auto
>
> topic
>
> creation is enabled on the broker, for example?
>
> Thanks,
> Dhruvil
>
> On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan <
>
> jolshan@confluent.io>
>
> wrote:
>
> Hello all,
>
> I'd like to start a discussion thread for KIP-487. This KIP plans to
> deprecate the current system of
>
> auto-creating
>
> topics
>
> through requests to the metadata and give the
>
> producer the
>
> ability to
>
> automatically create topics instead.
>
> More information can be found here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-487%3A+Automatic+Topic+Creation+on+Producer
>
> Thank you,
> Justine Olshan
>
>