You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Grant Henke <gh...@cloudera.com> on 2016/06/16 21:50:57 UTC
[VOTE] KIP-4 Create Topics Schema
I would like to initiate the voting process for the "KIP-4 Create Topics
Schema changes". This is not a vote for all of KIP-4, but specifically for
the create topics changes. I have included the exact changes below for
clarity:
>
> Create Topics Request (KAFKA-2945
> <https://issues.apache.org/jira/browse/KAFKA-2945>)
>
> CreateTopics Request (Version: 0) => [create_topic_requests] timeout
> create_topic_requests => topic num_partitions replication_factor [replica_assignment] [configs]
> topic => STRING
> num_partitions => INT32
> replication_factor => INT16
> replica_assignment => partition_id [replicas]
> partition_id => INT32
> replicas => INT32
> configs => config_key config_value
> config_key => STRING
> config_value => STRING
> timeout => INT32
>
> CreateTopicsRequest is a batch request to initiate topic creation with
> either predefined or automatic replica assignment and optionally topic
> configuration.
>
> Request semantics:
>
> 1. Must be sent to the controller broker
> 2. If there are multiple instructions for the same topic in one
> request an InvalidRequestException will be logged on the broker and the
> client will be disconnected.
> - This is because the list of topics is modeled server side as a
> map with TopicName as the key
> 3. The principal must be authorized to the "Create" Operation on the
> "Cluster" resource to create topics.
> - Unauthorized requests will receive a ClusterAuthorizationException
> 4.
>
> Only one from ReplicaAssignment or (num_partitions + replication_factor
> ), can be defined in one instruction.
> - If both parameters are specified an InvalidRequestException will be
> logged on the broker and the client will be disconnected.
> - In the case ReplicaAssignment is defined number of partitions and
> replicas will be calculated from the supplied replica_assignment.
> - In the case of defined (num_partitions + replication_factor)
> replica assignment will be automatically generated by the server.
> - One or the other must be defined. The existing broker side auto
> create defaults will not be used
> (default.replication.factor, num.partitions). The client implementation can
> have defaults for these options when generating the messages.
> - The first replica in [replicas] is assumed to be the preferred
> leader. This matches current behavior elsewhere.
> 5. Setting a timeout > 0 will allow the request to block until the
> topic metadata is "complete" on the controller node.
> - Complete means the local topic metadata cache been completely
> populated and all partitions have leaders
> - The topic metadata is updated when the controller sends out
> update metadata requests to the brokers
> - If a timeout error occurs, the topic could still be created
> successfully at a later time. Its up to the client to query for the state
> at that point.
> 6. Setting a timeout <= 0 will validate arguments and trigger the
> create topics and return immediately.
> - This is essentially the fully asynchronous mode we have in the
> Zookeeper tools today.
> - The error code in the response will either contain an argument
> validation exception or a timeout exception. If you receive a timeout
> exception, because you asked for 0 timeout, you can assume the message was
> valid and the topic creation was triggered.
> 7. The request is not transactional.
> 1. If an error occurs on one topic, the others could still be
> created.
> 2. Errors are reported independently.
>
> QA:
>
> - Why is CreateTopicsRequest a batch request?
> - Scenarios where tools or admins want to create many topics should
> be able to with fewer requests
> - Example: MirrorMaker may want to create the topics downstream
> - What happens if some topics error immediately? Will it
> return immediately?
> - The request will block until all topics have either been created,
> errors, or the timeout has been hit
> - There is no "short circuiting" where 1 error stops the other
> topics from being created
> - Why implement "partial blocking" instead of fully async or fully
> consistent?
> - See Cluster Consistent Blocking
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking>
> below
> - Why require the request to go to the controller?
> - The controller is responsible for the cluster metadata and its
> propagation
> - See Request Forwarding
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request>
> below
>
> Create Topics Response
>
>
>
> CreateTopics Response (Version: 0) => [topic_error_codes]
> topic_error_codes => topic error_code
> topic => STRING
> error_code => INT16
>
> CreateTopicsResponse contains a map between topic and topic creation
> result error code (see New Protocol Errors
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors>
> ).
>
The KIP is available here for reference (linked to the Create Topics schema
section):
*https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)>*
A pull request is available implementing the proposed changes here:
https://github.com/apache/kafka/pull/1489
Here is a link to the past discussion on the mailing list:
*http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
<http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema>*
Thank you,
Grant
--
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Grant Henke <gh...@cloudera.com>.
Thanks to all who voted. The KIP-4 Create Topics changes passed with +4
(binding), and +4 (non-binding).
There is a patch available for review here:
https://github.com/apache/kafka/pull/1489
On Tue, Jun 21, 2016 at 1:11 AM, Manikumar Reddy <ma...@gmail.com>
wrote:
> +1 (non-binding)
>
> On Tue, Jun 21, 2016 at 9:16 AM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > +1 (binding) and thanks for the work on this Grant!
> >
> > -Ewen
> >
> > On Mon, Jun 20, 2016 at 12:18 PM, Gwen Shapira <gw...@confluent.io>
> wrote:
> >
> > > +1 (binding)
> > >
> > > On Mon, Jun 20, 2016 at 12:13 PM, Tom Crayford <tc...@heroku.com>
> > > wrote:
> > > > +1 (non-binding)
> > > >
> > > > On Mon, Jun 20, 2016 at 8:07 PM, Harsha <ka...@harsha.io> wrote:
> > > >
> > > >> +1 (binding)
> > > >> -Harsha
> > > >>
> > > >> On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote:
> > > >> > +1 (binding)
> > > >> >
> > > >> > On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers <
> dana.powers@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > +1 -- thanks for the update
> > > >> > >
> > > >> > > On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke <
> > ghenke@cloudera.com>
> > > >> wrote:
> > > >> > > > I have update the patch and wiki based on the feedback in the
> > > >> discussion
> > > >> > > > thread. The only change is that instead of logging and
> > > disconnecting
> > > >> in
> > > >> > > the
> > > >> > > > case of invalid messages (duplicate topics or both arguments)
> we
> > > now
> > > >> > > return
> > > >> > > > and InvalidRequest error back to the client for that topic.
> > > >> > > >
> > > >> > > > I would like to restart the vote now including that change. If
> > you
> > > >> have
> > > >> > > > already voted, please revote in this thread.
> > > >> > > >
> > > >> > > > Thank you,
> > > >> > > > Grant
> > > >> > > >
> > > >> > > > On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> > > >> > > ewen@confluent.io>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > >> Don't necessarily want to add noise here, but I'm -1 based on
> > the
> > > >> > > >> disconnect part. See discussion in other thread. (I'm +1
> > > otherwise,
> > > >> and
> > > >> > > >> happy to have my vote applied assuming we clean up that one
> > > issue.)
> > > >> > > >>
> > > >> > > >> -Ewen
> > > >> > > >>
> > > >> > > >> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io>
> > wrote:
> > > >> > > >>
> > > >> > > >> > +1 (binding)
> > > >> > > >> > Thanks,
> > > >> > > >> > Harsha
> > > >> > > >> >
> > > >> > > >> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> > > >> > > >> > > +1.
> > > >> > > >> > >
> > > >> > > >> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <
> > > ismael@juma.me.uk
> > > >> >
> > > >> > > >> wrote:
> > > >> > > >> > >
> > > >> > > >> > > > +1 (binding)
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <
> > > >> > > ghenke@cloudera.com>
> > > >> > > >> > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > > I would like to initiate the voting process for the
> > > "KIP-4
> > > >> > > Create
> > > >> > > >> > Topics
> > > >> > > >> > > > > Schema changes". This is not a vote for all of KIP-4,
> > but
> > > >> > > >> > specifically
> > > >> > > >> > > > for
> > > >> > > >> > > > > the create topics changes. I have included the exact
> > > changes
> > > >> > > below
> > > >> > > >> > for
> > > >> > > >> > > > > clarity:
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > Create Topics Request (KAFKA-2945
> > > >> > > >> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945
> >)
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > CreateTopics Request (Version: 0) =>
> > > >> [create_topic_requests]
> > > >> > > >> > timeout
> > > >> > > >> > > > > > create_topic_requests => topic num_partitions
> > > >> > > >> replication_factor
> > > >> > > >> > > > > [replica_assignment] [configs]
> > > >> > > >> > > > > > topic => STRING
> > > >> > > >> > > > > > num_partitions => INT32
> > > >> > > >> > > > > > replication_factor => INT16
> > > >> > > >> > > > > > replica_assignment => partition_id [replicas]
> > > >> > > >> > > > > > partition_id => INT32
> > > >> > > >> > > > > > replicas => INT32
> > > >> > > >> > > > > > configs => config_key config_value
> > > >> > > >> > > > > > config_key => STRING
> > > >> > > >> > > > > > config_value => STRING
> > > >> > > >> > > > > > timeout => INT32
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > CreateTopicsRequest is a batch request to initiate
> > > topic
> > > >> > > creation
> > > >> > > >> > with
> > > >> > > >> > > > > > either predefined or automatic replica assignment
> and
> > > >> > > optionally
> > > >> > > >> > topic
> > > >> > > >> > > > > > configuration.
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > Request semantics:
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > 1. Must be sent to the controller broker
> > > >> > > >> > > > > > 2. If there are multiple instructions for the
> same
> > > >> topic in
> > > >> > > >> one
> > > >> > > >> > > > > > request an InvalidRequestException will be
> logged
> > on
> > > >> the
> > > >> > > >> broker
> > > >> > > >> > and
> > > >> > > >> > > > > the
> > > >> > > >> > > > > > client will be disconnected.
> > > >> > > >> > > > > > - This is because the list of topics is
> modeled
> > > >> server
> > > >> > > side
> > > >> > > >> > as a
> > > >> > > >> > > > > > map with TopicName as the key
> > > >> > > >> > > > > > 3. The principal must be authorized to the
> > "Create"
> > > >> > > Operation
> > > >> > > >> > on the
> > > >> > > >> > > > > > "Cluster" resource to create topics.
> > > >> > > >> > > > > > - Unauthorized requests will receive a
> > > >> > > >> > > > > ClusterAuthorizationException
> > > >> > > >> > > > > > 4.
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > Only one from ReplicaAssignment or
> > (num_partitions +
> > > >> > > >> > > > > replication_factor
> > > >> > > >> > > > > > ), can be defined in one instruction.
> > > >> > > >> > > > > > - If both parameters are specified an
> > > >> > > InvalidRequestException
> > > >> > > >> > will
> > > >> > > >> > > > be
> > > >> > > >> > > > > > logged on the broker and the client will be
> > > >> > > disconnected.
> > > >> > > >> > > > > > - In the case ReplicaAssignment is defined
> > > number of
> > > >> > > >> > partitions
> > > >> > > >> > > > and
> > > >> > > >> > > > > > replicas will be calculated from the supplied
> > > >> > > >> > replica_assignment.
> > > >> > > >> > > > > > - In the case of defined (num_partitions +
> > > >> > > >> > replication_factor)
> > > >> > > >> > > > > > replica assignment will be automatically
> > > generated
> > > >> by
> > > >> > > the
> > > >> > > >> > server.
> > > >> > > >> > > > > > - One or the other must be defined. The
> > existing
> > > >> broker
> > > >> > > >> side
> > > >> > > >> > auto
> > > >> > > >> > > > > > create defaults will not be used
> > > >> > > >> > > > > > (default.replication.factor, num.partitions).
> > The
> > > >> client
> > > >> > > >> > > > > implementation can
> > > >> > > >> > > > > > have defaults for these options when
> generating
> > > the
> > > >> > > >> messages.
> > > >> > > >> > > > > > - The first replica in [replicas] is assumed
> to
> > > be
> > > >> the
> > > >> > > >> > preferred
> > > >> > > >> > > > > > leader. This matches current behavior
> > elsewhere.
> > > >> > > >> > > > > > 5. Setting a timeout > 0 will allow the request
> to
> > > >> block
> > > >> > > until
> > > >> > > >> > the
> > > >> > > >> > > > > > topic metadata is "complete" on the controller
> > node.
> > > >> > > >> > > > > > - Complete means the local topic metadata
> cache
> > > been
> > > >> > > >> > completely
> > > >> > > >> > > > > > populated and all partitions have leaders
> > > >> > > >> > > > > > - The topic metadata is updated when the
> > > >> controller
> > > >> > > >> sends
> > > >> > > >> > out
> > > >> > > >> > > > > > update metadata requests to the brokers
> > > >> > > >> > > > > > - If a timeout error occurs, the topic could
> > > still
> > > >> be
> > > >> > > >> created
> > > >> > > >> > > > > > successfully at a later time. Its up to the
> > > client
> > > >> to
> > > >> > > query
> > > >> > > >> > for
> > > >> > > >> > > > > the state
> > > >> > > >> > > > > > at that point.
> > > >> > > >> > > > > > 6. Setting a timeout <= 0 will validate
> arguments
> > > and
> > > >> > > trigger
> > > >> > > >> > the
> > > >> > > >> > > > > > create topics and return immediately.
> > > >> > > >> > > > > > - This is essentially the fully asynchronous
> > > mode we
> > > >> > > have
> > > >> > > >> in
> > > >> > > >> > the
> > > >> > > >> > > > > > Zookeeper tools today.
> > > >> > > >> > > > > > - The error code in the response will either
> > > >> contain an
> > > >> > > >> > argument
> > > >> > > >> > > > > > validation exception or a timeout exception.
> If
> > > you
> > > >> > > >> receive a
> > > >> > > >> > > > > timeout
> > > >> > > >> > > > > > exception, because you asked for 0 timeout,
> you
> > > can
> > > >> > > assume
> > > >> > > >> > the
> > > >> > > >> > > > > message was
> > > >> > > >> > > > > > valid and the topic creation was triggered.
> > > >> > > >> > > > > > 7. The request is not transactional.
> > > >> > > >> > > > > > 1. If an error occurs on one topic, the
> others
> > > could
> > > >> > > still
> > > >> > > >> be
> > > >> > > >> > > > > > created.
> > > >> > > >> > > > > > 2. Errors are reported independently.
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > QA:
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > - Why is CreateTopicsRequest a batch request?
> > > >> > > >> > > > > > - Scenarios where tools or admins want to
> > create
> > > >> many
> > > >> > > >> topics
> > > >> > > >> > > > should
> > > >> > > >> > > > > > be able to with fewer requests
> > > >> > > >> > > > > > - Example: MirrorMaker may want to create the
> > > topics
> > > >> > > >> > downstream
> > > >> > > >> > > > > > - What happens if some topics error immediately?
> > > Will
> > > >> it
> > > >> > > >> > > > > > return immediately?
> > > >> > > >> > > > > > - The request will block until all topics
> have
> > > >> either
> > > >> > > been
> > > >> > > >> > > > created,
> > > >> > > >> > > > > > errors, or the timeout has been hit
> > > >> > > >> > > > > > - There is no "short circuiting" where 1
> error
> > > >> stops the
> > > >> > > >> > other
> > > >> > > >> > > > > > topics from being created
> > > >> > > >> > > > > > - Why implement "partial blocking" instead of
> > fully
> > > >> async
> > > >> > > or
> > > >> > > >> > fully
> > > >> > > >> > > > > > consistent?
> > > >> > > >> > > > > > - See Cluster Consistent Blocking
> > > >> > > >> > > > > > <
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > below
> > > >> > > >> > > > > > - Why require the request to go to the
> controller?
> > > >> > > >> > > > > > - The controller is responsible for the
> cluster
> > > >> metadata
> > > >> > > >> and
> > > >> > > >> > its
> > > >> > > >> > > > > > propagation
> > > >> > > >> > > > > > - See Request Forwarding
> > > >> > > >> > > > > > <
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > below
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > Create Topics Response
> > > >> > > >> > > > > >
> > > >> > > >> > > > > >
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > CreateTopics Response (Version: 0) =>
> > > [topic_error_codes]
> > > >> > > >> > > > > > topic_error_codes => topic error_code
> > > >> > > >> > > > > > topic => STRING
> > > >> > > >> > > > > > error_code => INT16
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > CreateTopicsResponse contains a map between topic
> and
> > > >> topic
> > > >> > > >> > creation
> > > >> > > >> > > > > > result error code (see New Protocol Errors
> > > >> > > >> > > > > > <
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > ).
> > > >> > > >> > > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > > > The KIP is available here for reference (linked to
> the
> > > >> Create
> > > >> > > >> Topics
> > > >> > > >> > > > schema
> > > >> > > >> > > > > section):
> > > >> > > >> > > > > *
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > >> > > >> > > > > <
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > >> > > >> > > > > >*
> > > >> > > >> > > > >
> > > >> > > >> > > > > A pull request is available implementing the proposed
> > > >> changes
> > > >> > > here:
> > > >> > > >> > > > > https://github.com/apache/kafka/pull/1489
> > > >> > > >> > > > >
> > > >> > > >> > > > > Here is a link to the past discussion on the mailing
> > > list:
> > > >> > > >> > > > > *
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > >> > > >> > > > > <
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > >> > > >> > > > > >*
> > > >> > > >> > > > >
> > > >> > > >> > > > > Thank you,
> > > >> > > >> > > > > Grant
> > > >> > > >> > > > > --
> > > >> > > >> > > > > Grant Henke
> > > >> > > >> > > > > Software Engineer | Cloudera
> > > >> > > >> > > > > grant@cloudera.com | twitter.com/gchenke |
> > > >> > > >> > linkedin.com/in/granthenke
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > > --
> > > >> > > >> > > -- Guozhang
> > > >> > > >> >
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> --
> > > >> > > >> Thanks,
> > > >> > > >> Ewen
> > > >> > > >>
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > Grant Henke
> > > >> > > > Software Engineer | Cloudera
> > > >> > > > grant@cloudera.com | twitter.com/gchenke |
> > > >> linkedin.com/in/granthenke
> > > >> > >
> > > >>
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>
--
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Manikumar Reddy <ma...@gmail.com>.
+1 (non-binding)
On Tue, Jun 21, 2016 at 9:16 AM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> +1 (binding) and thanks for the work on this Grant!
>
> -Ewen
>
> On Mon, Jun 20, 2016 at 12:18 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > +1 (binding)
> >
> > On Mon, Jun 20, 2016 at 12:13 PM, Tom Crayford <tc...@heroku.com>
> > wrote:
> > > +1 (non-binding)
> > >
> > > On Mon, Jun 20, 2016 at 8:07 PM, Harsha <ka...@harsha.io> wrote:
> > >
> > >> +1 (binding)
> > >> -Harsha
> > >>
> > >> On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote:
> > >> > +1 (binding)
> > >> >
> > >> > On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers <dana.powers@gmail.com
> >
> > >> > wrote:
> > >> >
> > >> > > +1 -- thanks for the update
> > >> > >
> > >> > > On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke <
> ghenke@cloudera.com>
> > >> wrote:
> > >> > > > I have update the patch and wiki based on the feedback in the
> > >> discussion
> > >> > > > thread. The only change is that instead of logging and
> > disconnecting
> > >> in
> > >> > > the
> > >> > > > case of invalid messages (duplicate topics or both arguments) we
> > now
> > >> > > return
> > >> > > > and InvalidRequest error back to the client for that topic.
> > >> > > >
> > >> > > > I would like to restart the vote now including that change. If
> you
> > >> have
> > >> > > > already voted, please revote in this thread.
> > >> > > >
> > >> > > > Thank you,
> > >> > > > Grant
> > >> > > >
> > >> > > > On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> > >> > > ewen@confluent.io>
> > >> > > > wrote:
> > >> > > >
> > >> > > >> Don't necessarily want to add noise here, but I'm -1 based on
> the
> > >> > > >> disconnect part. See discussion in other thread. (I'm +1
> > otherwise,
> > >> and
> > >> > > >> happy to have my vote applied assuming we clean up that one
> > issue.)
> > >> > > >>
> > >> > > >> -Ewen
> > >> > > >>
> > >> > > >> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io>
> wrote:
> > >> > > >>
> > >> > > >> > +1 (binding)
> > >> > > >> > Thanks,
> > >> > > >> > Harsha
> > >> > > >> >
> > >> > > >> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> > >> > > >> > > +1.
> > >> > > >> > >
> > >> > > >> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <
> > ismael@juma.me.uk
> > >> >
> > >> > > >> wrote:
> > >> > > >> > >
> > >> > > >> > > > +1 (binding)
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <
> > >> > > ghenke@cloudera.com>
> > >> > > >> > wrote:
> > >> > > >> > > >
> > >> > > >> > > > > I would like to initiate the voting process for the
> > "KIP-4
> > >> > > Create
> > >> > > >> > Topics
> > >> > > >> > > > > Schema changes". This is not a vote for all of KIP-4,
> but
> > >> > > >> > specifically
> > >> > > >> > > > for
> > >> > > >> > > > > the create topics changes. I have included the exact
> > changes
> > >> > > below
> > >> > > >> > for
> > >> > > >> > > > > clarity:
> > >> > > >> > > > > >
> > >> > > >> > > > > > Create Topics Request (KAFKA-2945
> > >> > > >> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> > >> > > >> > > > > >
> > >> > > >> > > > > > CreateTopics Request (Version: 0) =>
> > >> [create_topic_requests]
> > >> > > >> > timeout
> > >> > > >> > > > > > create_topic_requests => topic num_partitions
> > >> > > >> replication_factor
> > >> > > >> > > > > [replica_assignment] [configs]
> > >> > > >> > > > > > topic => STRING
> > >> > > >> > > > > > num_partitions => INT32
> > >> > > >> > > > > > replication_factor => INT16
> > >> > > >> > > > > > replica_assignment => partition_id [replicas]
> > >> > > >> > > > > > partition_id => INT32
> > >> > > >> > > > > > replicas => INT32
> > >> > > >> > > > > > configs => config_key config_value
> > >> > > >> > > > > > config_key => STRING
> > >> > > >> > > > > > config_value => STRING
> > >> > > >> > > > > > timeout => INT32
> > >> > > >> > > > > >
> > >> > > >> > > > > > CreateTopicsRequest is a batch request to initiate
> > topic
> > >> > > creation
> > >> > > >> > with
> > >> > > >> > > > > > either predefined or automatic replica assignment and
> > >> > > optionally
> > >> > > >> > topic
> > >> > > >> > > > > > configuration.
> > >> > > >> > > > > >
> > >> > > >> > > > > > Request semantics:
> > >> > > >> > > > > >
> > >> > > >> > > > > > 1. Must be sent to the controller broker
> > >> > > >> > > > > > 2. If there are multiple instructions for the same
> > >> topic in
> > >> > > >> one
> > >> > > >> > > > > > request an InvalidRequestException will be logged
> on
> > >> the
> > >> > > >> broker
> > >> > > >> > and
> > >> > > >> > > > > the
> > >> > > >> > > > > > client will be disconnected.
> > >> > > >> > > > > > - This is because the list of topics is modeled
> > >> server
> > >> > > side
> > >> > > >> > as a
> > >> > > >> > > > > > map with TopicName as the key
> > >> > > >> > > > > > 3. The principal must be authorized to the
> "Create"
> > >> > > Operation
> > >> > > >> > on the
> > >> > > >> > > > > > "Cluster" resource to create topics.
> > >> > > >> > > > > > - Unauthorized requests will receive a
> > >> > > >> > > > > ClusterAuthorizationException
> > >> > > >> > > > > > 4.
> > >> > > >> > > > > >
> > >> > > >> > > > > > Only one from ReplicaAssignment or
> (num_partitions +
> > >> > > >> > > > > replication_factor
> > >> > > >> > > > > > ), can be defined in one instruction.
> > >> > > >> > > > > > - If both parameters are specified an
> > >> > > InvalidRequestException
> > >> > > >> > will
> > >> > > >> > > > be
> > >> > > >> > > > > > logged on the broker and the client will be
> > >> > > disconnected.
> > >> > > >> > > > > > - In the case ReplicaAssignment is defined
> > number of
> > >> > > >> > partitions
> > >> > > >> > > > and
> > >> > > >> > > > > > replicas will be calculated from the supplied
> > >> > > >> > replica_assignment.
> > >> > > >> > > > > > - In the case of defined (num_partitions +
> > >> > > >> > replication_factor)
> > >> > > >> > > > > > replica assignment will be automatically
> > generated
> > >> by
> > >> > > the
> > >> > > >> > server.
> > >> > > >> > > > > > - One or the other must be defined. The
> existing
> > >> broker
> > >> > > >> side
> > >> > > >> > auto
> > >> > > >> > > > > > create defaults will not be used
> > >> > > >> > > > > > (default.replication.factor, num.partitions).
> The
> > >> client
> > >> > > >> > > > > implementation can
> > >> > > >> > > > > > have defaults for these options when generating
> > the
> > >> > > >> messages.
> > >> > > >> > > > > > - The first replica in [replicas] is assumed to
> > be
> > >> the
> > >> > > >> > preferred
> > >> > > >> > > > > > leader. This matches current behavior
> elsewhere.
> > >> > > >> > > > > > 5. Setting a timeout > 0 will allow the request to
> > >> block
> > >> > > until
> > >> > > >> > the
> > >> > > >> > > > > > topic metadata is "complete" on the controller
> node.
> > >> > > >> > > > > > - Complete means the local topic metadata cache
> > been
> > >> > > >> > completely
> > >> > > >> > > > > > populated and all partitions have leaders
> > >> > > >> > > > > > - The topic metadata is updated when the
> > >> controller
> > >> > > >> sends
> > >> > > >> > out
> > >> > > >> > > > > > update metadata requests to the brokers
> > >> > > >> > > > > > - If a timeout error occurs, the topic could
> > still
> > >> be
> > >> > > >> created
> > >> > > >> > > > > > successfully at a later time. Its up to the
> > client
> > >> to
> > >> > > query
> > >> > > >> > for
> > >> > > >> > > > > the state
> > >> > > >> > > > > > at that point.
> > >> > > >> > > > > > 6. Setting a timeout <= 0 will validate arguments
> > and
> > >> > > trigger
> > >> > > >> > the
> > >> > > >> > > > > > create topics and return immediately.
> > >> > > >> > > > > > - This is essentially the fully asynchronous
> > mode we
> > >> > > have
> > >> > > >> in
> > >> > > >> > the
> > >> > > >> > > > > > Zookeeper tools today.
> > >> > > >> > > > > > - The error code in the response will either
> > >> contain an
> > >> > > >> > argument
> > >> > > >> > > > > > validation exception or a timeout exception. If
> > you
> > >> > > >> receive a
> > >> > > >> > > > > timeout
> > >> > > >> > > > > > exception, because you asked for 0 timeout, you
> > can
> > >> > > assume
> > >> > > >> > the
> > >> > > >> > > > > message was
> > >> > > >> > > > > > valid and the topic creation was triggered.
> > >> > > >> > > > > > 7. The request is not transactional.
> > >> > > >> > > > > > 1. If an error occurs on one topic, the others
> > could
> > >> > > still
> > >> > > >> be
> > >> > > >> > > > > > created.
> > >> > > >> > > > > > 2. Errors are reported independently.
> > >> > > >> > > > > >
> > >> > > >> > > > > > QA:
> > >> > > >> > > > > >
> > >> > > >> > > > > > - Why is CreateTopicsRequest a batch request?
> > >> > > >> > > > > > - Scenarios where tools or admins want to
> create
> > >> many
> > >> > > >> topics
> > >> > > >> > > > should
> > >> > > >> > > > > > be able to with fewer requests
> > >> > > >> > > > > > - Example: MirrorMaker may want to create the
> > topics
> > >> > > >> > downstream
> > >> > > >> > > > > > - What happens if some topics error immediately?
> > Will
> > >> it
> > >> > > >> > > > > > return immediately?
> > >> > > >> > > > > > - The request will block until all topics have
> > >> either
> > >> > > been
> > >> > > >> > > > created,
> > >> > > >> > > > > > errors, or the timeout has been hit
> > >> > > >> > > > > > - There is no "short circuiting" where 1 error
> > >> stops the
> > >> > > >> > other
> > >> > > >> > > > > > topics from being created
> > >> > > >> > > > > > - Why implement "partial blocking" instead of
> fully
> > >> async
> > >> > > or
> > >> > > >> > fully
> > >> > > >> > > > > > consistent?
> > >> > > >> > > > > > - See Cluster Consistent Blocking
> > >> > > >> > > > > > <
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> > >> > > >> > > > > >
> > >> > > >> > > > > > below
> > >> > > >> > > > > > - Why require the request to go to the controller?
> > >> > > >> > > > > > - The controller is responsible for the cluster
> > >> metadata
> > >> > > >> and
> > >> > > >> > its
> > >> > > >> > > > > > propagation
> > >> > > >> > > > > > - See Request Forwarding
> > >> > > >> > > > > > <
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> > >> > > >> > > > > >
> > >> > > >> > > > > > below
> > >> > > >> > > > > >
> > >> > > >> > > > > > Create Topics Response
> > >> > > >> > > > > >
> > >> > > >> > > > > >
> > >> > > >> > > > > >
> > >> > > >> > > > > > CreateTopics Response (Version: 0) =>
> > [topic_error_codes]
> > >> > > >> > > > > > topic_error_codes => topic error_code
> > >> > > >> > > > > > topic => STRING
> > >> > > >> > > > > > error_code => INT16
> > >> > > >> > > > > >
> > >> > > >> > > > > > CreateTopicsResponse contains a map between topic and
> > >> topic
> > >> > > >> > creation
> > >> > > >> > > > > > result error code (see New Protocol Errors
> > >> > > >> > > > > > <
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> > >> > > >> > > > > >
> > >> > > >> > > > > > ).
> > >> > > >> > > > > >
> > >> > > >> > > > >
> > >> > > >> > > > > The KIP is available here for reference (linked to the
> > >> Create
> > >> > > >> Topics
> > >> > > >> > > > schema
> > >> > > >> > > > > section):
> > >> > > >> > > > > *
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > >> > > >> > > > > <
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > >> > > >> > > > > >*
> > >> > > >> > > > >
> > >> > > >> > > > > A pull request is available implementing the proposed
> > >> changes
> > >> > > here:
> > >> > > >> > > > > https://github.com/apache/kafka/pull/1489
> > >> > > >> > > > >
> > >> > > >> > > > > Here is a link to the past discussion on the mailing
> > list:
> > >> > > >> > > > > *
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > >> > > >> > > > > <
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > >> > > >> > > > > >*
> > >> > > >> > > > >
> > >> > > >> > > > > Thank you,
> > >> > > >> > > > > Grant
> > >> > > >> > > > > --
> > >> > > >> > > > > Grant Henke
> > >> > > >> > > > > Software Engineer | Cloudera
> > >> > > >> > > > > grant@cloudera.com | twitter.com/gchenke |
> > >> > > >> > linkedin.com/in/granthenke
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > > --
> > >> > > >> > > -- Guozhang
> > >> > > >> >
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> --
> > >> > > >> Thanks,
> > >> > > >> Ewen
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > Grant Henke
> > >> > > > Software Engineer | Cloudera
> > >> > > > grant@cloudera.com | twitter.com/gchenke |
> > >> linkedin.com/in/granthenke
> > >> > >
> > >>
> >
>
>
>
> --
> Thanks,
> Ewen
>
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
+1 (binding) and thanks for the work on this Grant!
-Ewen
On Mon, Jun 20, 2016 at 12:18 PM, Gwen Shapira <gw...@confluent.io> wrote:
> +1 (binding)
>
> On Mon, Jun 20, 2016 at 12:13 PM, Tom Crayford <tc...@heroku.com>
> wrote:
> > +1 (non-binding)
> >
> > On Mon, Jun 20, 2016 at 8:07 PM, Harsha <ka...@harsha.io> wrote:
> >
> >> +1 (binding)
> >> -Harsha
> >>
> >> On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote:
> >> > +1 (binding)
> >> >
> >> > On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers <da...@gmail.com>
> >> > wrote:
> >> >
> >> > > +1 -- thanks for the update
> >> > >
> >> > > On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke <gh...@cloudera.com>
> >> wrote:
> >> > > > I have update the patch and wiki based on the feedback in the
> >> discussion
> >> > > > thread. The only change is that instead of logging and
> disconnecting
> >> in
> >> > > the
> >> > > > case of invalid messages (duplicate topics or both arguments) we
> now
> >> > > return
> >> > > > and InvalidRequest error back to the client for that topic.
> >> > > >
> >> > > > I would like to restart the vote now including that change. If you
> >> have
> >> > > > already voted, please revote in this thread.
> >> > > >
> >> > > > Thank you,
> >> > > > Grant
> >> > > >
> >> > > > On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> >> > > ewen@confluent.io>
> >> > > > wrote:
> >> > > >
> >> > > >> Don't necessarily want to add noise here, but I'm -1 based on the
> >> > > >> disconnect part. See discussion in other thread. (I'm +1
> otherwise,
> >> and
> >> > > >> happy to have my vote applied assuming we clean up that one
> issue.)
> >> > > >>
> >> > > >> -Ewen
> >> > > >>
> >> > > >> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io> wrote:
> >> > > >>
> >> > > >> > +1 (binding)
> >> > > >> > Thanks,
> >> > > >> > Harsha
> >> > > >> >
> >> > > >> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> >> > > >> > > +1.
> >> > > >> > >
> >> > > >> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <
> ismael@juma.me.uk
> >> >
> >> > > >> wrote:
> >> > > >> > >
> >> > > >> > > > +1 (binding)
> >> > > >> > > >
> >> > > >> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <
> >> > > ghenke@cloudera.com>
> >> > > >> > wrote:
> >> > > >> > > >
> >> > > >> > > > > I would like to initiate the voting process for the
> "KIP-4
> >> > > Create
> >> > > >> > Topics
> >> > > >> > > > > Schema changes". This is not a vote for all of KIP-4, but
> >> > > >> > specifically
> >> > > >> > > > for
> >> > > >> > > > > the create topics changes. I have included the exact
> changes
> >> > > below
> >> > > >> > for
> >> > > >> > > > > clarity:
> >> > > >> > > > > >
> >> > > >> > > > > > Create Topics Request (KAFKA-2945
> >> > > >> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> >> > > >> > > > > >
> >> > > >> > > > > > CreateTopics Request (Version: 0) =>
> >> [create_topic_requests]
> >> > > >> > timeout
> >> > > >> > > > > > create_topic_requests => topic num_partitions
> >> > > >> replication_factor
> >> > > >> > > > > [replica_assignment] [configs]
> >> > > >> > > > > > topic => STRING
> >> > > >> > > > > > num_partitions => INT32
> >> > > >> > > > > > replication_factor => INT16
> >> > > >> > > > > > replica_assignment => partition_id [replicas]
> >> > > >> > > > > > partition_id => INT32
> >> > > >> > > > > > replicas => INT32
> >> > > >> > > > > > configs => config_key config_value
> >> > > >> > > > > > config_key => STRING
> >> > > >> > > > > > config_value => STRING
> >> > > >> > > > > > timeout => INT32
> >> > > >> > > > > >
> >> > > >> > > > > > CreateTopicsRequest is a batch request to initiate
> topic
> >> > > creation
> >> > > >> > with
> >> > > >> > > > > > either predefined or automatic replica assignment and
> >> > > optionally
> >> > > >> > topic
> >> > > >> > > > > > configuration.
> >> > > >> > > > > >
> >> > > >> > > > > > Request semantics:
> >> > > >> > > > > >
> >> > > >> > > > > > 1. Must be sent to the controller broker
> >> > > >> > > > > > 2. If there are multiple instructions for the same
> >> topic in
> >> > > >> one
> >> > > >> > > > > > request an InvalidRequestException will be logged on
> >> the
> >> > > >> broker
> >> > > >> > and
> >> > > >> > > > > the
> >> > > >> > > > > > client will be disconnected.
> >> > > >> > > > > > - This is because the list of topics is modeled
> >> server
> >> > > side
> >> > > >> > as a
> >> > > >> > > > > > map with TopicName as the key
> >> > > >> > > > > > 3. The principal must be authorized to the "Create"
> >> > > Operation
> >> > > >> > on the
> >> > > >> > > > > > "Cluster" resource to create topics.
> >> > > >> > > > > > - Unauthorized requests will receive a
> >> > > >> > > > > ClusterAuthorizationException
> >> > > >> > > > > > 4.
> >> > > >> > > > > >
> >> > > >> > > > > > Only one from ReplicaAssignment or (num_partitions +
> >> > > >> > > > > replication_factor
> >> > > >> > > > > > ), can be defined in one instruction.
> >> > > >> > > > > > - If both parameters are specified an
> >> > > InvalidRequestException
> >> > > >> > will
> >> > > >> > > > be
> >> > > >> > > > > > logged on the broker and the client will be
> >> > > disconnected.
> >> > > >> > > > > > - In the case ReplicaAssignment is defined
> number of
> >> > > >> > partitions
> >> > > >> > > > and
> >> > > >> > > > > > replicas will be calculated from the supplied
> >> > > >> > replica_assignment.
> >> > > >> > > > > > - In the case of defined (num_partitions +
> >> > > >> > replication_factor)
> >> > > >> > > > > > replica assignment will be automatically
> generated
> >> by
> >> > > the
> >> > > >> > server.
> >> > > >> > > > > > - One or the other must be defined. The existing
> >> broker
> >> > > >> side
> >> > > >> > auto
> >> > > >> > > > > > create defaults will not be used
> >> > > >> > > > > > (default.replication.factor, num.partitions). The
> >> client
> >> > > >> > > > > implementation can
> >> > > >> > > > > > have defaults for these options when generating
> the
> >> > > >> messages.
> >> > > >> > > > > > - The first replica in [replicas] is assumed to
> be
> >> the
> >> > > >> > preferred
> >> > > >> > > > > > leader. This matches current behavior elsewhere.
> >> > > >> > > > > > 5. Setting a timeout > 0 will allow the request to
> >> block
> >> > > until
> >> > > >> > the
> >> > > >> > > > > > topic metadata is "complete" on the controller node.
> >> > > >> > > > > > - Complete means the local topic metadata cache
> been
> >> > > >> > completely
> >> > > >> > > > > > populated and all partitions have leaders
> >> > > >> > > > > > - The topic metadata is updated when the
> >> controller
> >> > > >> sends
> >> > > >> > out
> >> > > >> > > > > > update metadata requests to the brokers
> >> > > >> > > > > > - If a timeout error occurs, the topic could
> still
> >> be
> >> > > >> created
> >> > > >> > > > > > successfully at a later time. Its up to the
> client
> >> to
> >> > > query
> >> > > >> > for
> >> > > >> > > > > the state
> >> > > >> > > > > > at that point.
> >> > > >> > > > > > 6. Setting a timeout <= 0 will validate arguments
> and
> >> > > trigger
> >> > > >> > the
> >> > > >> > > > > > create topics and return immediately.
> >> > > >> > > > > > - This is essentially the fully asynchronous
> mode we
> >> > > have
> >> > > >> in
> >> > > >> > the
> >> > > >> > > > > > Zookeeper tools today.
> >> > > >> > > > > > - The error code in the response will either
> >> contain an
> >> > > >> > argument
> >> > > >> > > > > > validation exception or a timeout exception. If
> you
> >> > > >> receive a
> >> > > >> > > > > timeout
> >> > > >> > > > > > exception, because you asked for 0 timeout, you
> can
> >> > > assume
> >> > > >> > the
> >> > > >> > > > > message was
> >> > > >> > > > > > valid and the topic creation was triggered.
> >> > > >> > > > > > 7. The request is not transactional.
> >> > > >> > > > > > 1. If an error occurs on one topic, the others
> could
> >> > > still
> >> > > >> be
> >> > > >> > > > > > created.
> >> > > >> > > > > > 2. Errors are reported independently.
> >> > > >> > > > > >
> >> > > >> > > > > > QA:
> >> > > >> > > > > >
> >> > > >> > > > > > - Why is CreateTopicsRequest a batch request?
> >> > > >> > > > > > - Scenarios where tools or admins want to create
> >> many
> >> > > >> topics
> >> > > >> > > > should
> >> > > >> > > > > > be able to with fewer requests
> >> > > >> > > > > > - Example: MirrorMaker may want to create the
> topics
> >> > > >> > downstream
> >> > > >> > > > > > - What happens if some topics error immediately?
> Will
> >> it
> >> > > >> > > > > > return immediately?
> >> > > >> > > > > > - The request will block until all topics have
> >> either
> >> > > been
> >> > > >> > > > created,
> >> > > >> > > > > > errors, or the timeout has been hit
> >> > > >> > > > > > - There is no "short circuiting" where 1 error
> >> stops the
> >> > > >> > other
> >> > > >> > > > > > topics from being created
> >> > > >> > > > > > - Why implement "partial blocking" instead of fully
> >> async
> >> > > or
> >> > > >> > fully
> >> > > >> > > > > > consistent?
> >> > > >> > > > > > - See Cluster Consistent Blocking
> >> > > >> > > > > > <
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> >> > > >> > > > > >
> >> > > >> > > > > > below
> >> > > >> > > > > > - Why require the request to go to the controller?
> >> > > >> > > > > > - The controller is responsible for the cluster
> >> metadata
> >> > > >> and
> >> > > >> > its
> >> > > >> > > > > > propagation
> >> > > >> > > > > > - See Request Forwarding
> >> > > >> > > > > > <
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> >> > > >> > > > > >
> >> > > >> > > > > > below
> >> > > >> > > > > >
> >> > > >> > > > > > Create Topics Response
> >> > > >> > > > > >
> >> > > >> > > > > >
> >> > > >> > > > > >
> >> > > >> > > > > > CreateTopics Response (Version: 0) =>
> [topic_error_codes]
> >> > > >> > > > > > topic_error_codes => topic error_code
> >> > > >> > > > > > topic => STRING
> >> > > >> > > > > > error_code => INT16
> >> > > >> > > > > >
> >> > > >> > > > > > CreateTopicsResponse contains a map between topic and
> >> topic
> >> > > >> > creation
> >> > > >> > > > > > result error code (see New Protocol Errors
> >> > > >> > > > > > <
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> >> > > >> > > > > >
> >> > > >> > > > > > ).
> >> > > >> > > > > >
> >> > > >> > > > >
> >> > > >> > > > > The KIP is available here for reference (linked to the
> >> Create
> >> > > >> Topics
> >> > > >> > > > schema
> >> > > >> > > > > section):
> >> > > >> > > > > *
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> >> > > >> > > > > <
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> >> > > >> > > > > >*
> >> > > >> > > > >
> >> > > >> > > > > A pull request is available implementing the proposed
> >> changes
> >> > > here:
> >> > > >> > > > > https://github.com/apache/kafka/pull/1489
> >> > > >> > > > >
> >> > > >> > > > > Here is a link to the past discussion on the mailing
> list:
> >> > > >> > > > > *
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> >> > > >> > > > > <
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> >> > > >> > > > > >*
> >> > > >> > > > >
> >> > > >> > > > > Thank you,
> >> > > >> > > > > Grant
> >> > > >> > > > > --
> >> > > >> > > > > Grant Henke
> >> > > >> > > > > Software Engineer | Cloudera
> >> > > >> > > > > grant@cloudera.com | twitter.com/gchenke |
> >> > > >> > linkedin.com/in/granthenke
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > --
> >> > > >> > > -- Guozhang
> >> > > >> >
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >> Thanks,
> >> > > >> Ewen
> >> > > >>
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Grant Henke
> >> > > > Software Engineer | Cloudera
> >> > > > grant@cloudera.com | twitter.com/gchenke |
> >> linkedin.com/in/granthenke
> >> > >
> >>
>
--
Thanks,
Ewen
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Gwen Shapira <gw...@confluent.io>.
+1 (binding)
On Mon, Jun 20, 2016 at 12:13 PM, Tom Crayford <tc...@heroku.com> wrote:
> +1 (non-binding)
>
> On Mon, Jun 20, 2016 at 8:07 PM, Harsha <ka...@harsha.io> wrote:
>
>> +1 (binding)
>> -Harsha
>>
>> On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote:
>> > +1 (binding)
>> >
>> > On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers <da...@gmail.com>
>> > wrote:
>> >
>> > > +1 -- thanks for the update
>> > >
>> > > On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke <gh...@cloudera.com>
>> wrote:
>> > > > I have update the patch and wiki based on the feedback in the
>> discussion
>> > > > thread. The only change is that instead of logging and disconnecting
>> in
>> > > the
>> > > > case of invalid messages (duplicate topics or both arguments) we now
>> > > return
>> > > > and InvalidRequest error back to the client for that topic.
>> > > >
>> > > > I would like to restart the vote now including that change. If you
>> have
>> > > > already voted, please revote in this thread.
>> > > >
>> > > > Thank you,
>> > > > Grant
>> > > >
>> > > > On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <
>> > > ewen@confluent.io>
>> > > > wrote:
>> > > >
>> > > >> Don't necessarily want to add noise here, but I'm -1 based on the
>> > > >> disconnect part. See discussion in other thread. (I'm +1 otherwise,
>> and
>> > > >> happy to have my vote applied assuming we clean up that one issue.)
>> > > >>
>> > > >> -Ewen
>> > > >>
>> > > >> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io> wrote:
>> > > >>
>> > > >> > +1 (binding)
>> > > >> > Thanks,
>> > > >> > Harsha
>> > > >> >
>> > > >> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
>> > > >> > > +1.
>> > > >> > >
>> > > >> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <ismael@juma.me.uk
>> >
>> > > >> wrote:
>> > > >> > >
>> > > >> > > > +1 (binding)
>> > > >> > > >
>> > > >> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <
>> > > ghenke@cloudera.com>
>> > > >> > wrote:
>> > > >> > > >
>> > > >> > > > > I would like to initiate the voting process for the "KIP-4
>> > > Create
>> > > >> > Topics
>> > > >> > > > > Schema changes". This is not a vote for all of KIP-4, but
>> > > >> > specifically
>> > > >> > > > for
>> > > >> > > > > the create topics changes. I have included the exact changes
>> > > below
>> > > >> > for
>> > > >> > > > > clarity:
>> > > >> > > > > >
>> > > >> > > > > > Create Topics Request (KAFKA-2945
>> > > >> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
>> > > >> > > > > >
>> > > >> > > > > > CreateTopics Request (Version: 0) =>
>> [create_topic_requests]
>> > > >> > timeout
>> > > >> > > > > > create_topic_requests => topic num_partitions
>> > > >> replication_factor
>> > > >> > > > > [replica_assignment] [configs]
>> > > >> > > > > > topic => STRING
>> > > >> > > > > > num_partitions => INT32
>> > > >> > > > > > replication_factor => INT16
>> > > >> > > > > > replica_assignment => partition_id [replicas]
>> > > >> > > > > > partition_id => INT32
>> > > >> > > > > > replicas => INT32
>> > > >> > > > > > configs => config_key config_value
>> > > >> > > > > > config_key => STRING
>> > > >> > > > > > config_value => STRING
>> > > >> > > > > > timeout => INT32
>> > > >> > > > > >
>> > > >> > > > > > CreateTopicsRequest is a batch request to initiate topic
>> > > creation
>> > > >> > with
>> > > >> > > > > > either predefined or automatic replica assignment and
>> > > optionally
>> > > >> > topic
>> > > >> > > > > > configuration.
>> > > >> > > > > >
>> > > >> > > > > > Request semantics:
>> > > >> > > > > >
>> > > >> > > > > > 1. Must be sent to the controller broker
>> > > >> > > > > > 2. If there are multiple instructions for the same
>> topic in
>> > > >> one
>> > > >> > > > > > request an InvalidRequestException will be logged on
>> the
>> > > >> broker
>> > > >> > and
>> > > >> > > > > the
>> > > >> > > > > > client will be disconnected.
>> > > >> > > > > > - This is because the list of topics is modeled
>> server
>> > > side
>> > > >> > as a
>> > > >> > > > > > map with TopicName as the key
>> > > >> > > > > > 3. The principal must be authorized to the "Create"
>> > > Operation
>> > > >> > on the
>> > > >> > > > > > "Cluster" resource to create topics.
>> > > >> > > > > > - Unauthorized requests will receive a
>> > > >> > > > > ClusterAuthorizationException
>> > > >> > > > > > 4.
>> > > >> > > > > >
>> > > >> > > > > > Only one from ReplicaAssignment or (num_partitions +
>> > > >> > > > > replication_factor
>> > > >> > > > > > ), can be defined in one instruction.
>> > > >> > > > > > - If both parameters are specified an
>> > > InvalidRequestException
>> > > >> > will
>> > > >> > > > be
>> > > >> > > > > > logged on the broker and the client will be
>> > > disconnected.
>> > > >> > > > > > - In the case ReplicaAssignment is defined number of
>> > > >> > partitions
>> > > >> > > > and
>> > > >> > > > > > replicas will be calculated from the supplied
>> > > >> > replica_assignment.
>> > > >> > > > > > - In the case of defined (num_partitions +
>> > > >> > replication_factor)
>> > > >> > > > > > replica assignment will be automatically generated
>> by
>> > > the
>> > > >> > server.
>> > > >> > > > > > - One or the other must be defined. The existing
>> broker
>> > > >> side
>> > > >> > auto
>> > > >> > > > > > create defaults will not be used
>> > > >> > > > > > (default.replication.factor, num.partitions). The
>> client
>> > > >> > > > > implementation can
>> > > >> > > > > > have defaults for these options when generating the
>> > > >> messages.
>> > > >> > > > > > - The first replica in [replicas] is assumed to be
>> the
>> > > >> > preferred
>> > > >> > > > > > leader. This matches current behavior elsewhere.
>> > > >> > > > > > 5. Setting a timeout > 0 will allow the request to
>> block
>> > > until
>> > > >> > the
>> > > >> > > > > > topic metadata is "complete" on the controller node.
>> > > >> > > > > > - Complete means the local topic metadata cache been
>> > > >> > completely
>> > > >> > > > > > populated and all partitions have leaders
>> > > >> > > > > > - The topic metadata is updated when the
>> controller
>> > > >> sends
>> > > >> > out
>> > > >> > > > > > update metadata requests to the brokers
>> > > >> > > > > > - If a timeout error occurs, the topic could still
>> be
>> > > >> created
>> > > >> > > > > > successfully at a later time. Its up to the client
>> to
>> > > query
>> > > >> > for
>> > > >> > > > > the state
>> > > >> > > > > > at that point.
>> > > >> > > > > > 6. Setting a timeout <= 0 will validate arguments and
>> > > trigger
>> > > >> > the
>> > > >> > > > > > create topics and return immediately.
>> > > >> > > > > > - This is essentially the fully asynchronous mode we
>> > > have
>> > > >> in
>> > > >> > the
>> > > >> > > > > > Zookeeper tools today.
>> > > >> > > > > > - The error code in the response will either
>> contain an
>> > > >> > argument
>> > > >> > > > > > validation exception or a timeout exception. If you
>> > > >> receive a
>> > > >> > > > > timeout
>> > > >> > > > > > exception, because you asked for 0 timeout, you can
>> > > assume
>> > > >> > the
>> > > >> > > > > message was
>> > > >> > > > > > valid and the topic creation was triggered.
>> > > >> > > > > > 7. The request is not transactional.
>> > > >> > > > > > 1. If an error occurs on one topic, the others could
>> > > still
>> > > >> be
>> > > >> > > > > > created.
>> > > >> > > > > > 2. Errors are reported independently.
>> > > >> > > > > >
>> > > >> > > > > > QA:
>> > > >> > > > > >
>> > > >> > > > > > - Why is CreateTopicsRequest a batch request?
>> > > >> > > > > > - Scenarios where tools or admins want to create
>> many
>> > > >> topics
>> > > >> > > > should
>> > > >> > > > > > be able to with fewer requests
>> > > >> > > > > > - Example: MirrorMaker may want to create the topics
>> > > >> > downstream
>> > > >> > > > > > - What happens if some topics error immediately? Will
>> it
>> > > >> > > > > > return immediately?
>> > > >> > > > > > - The request will block until all topics have
>> either
>> > > been
>> > > >> > > > created,
>> > > >> > > > > > errors, or the timeout has been hit
>> > > >> > > > > > - There is no "short circuiting" where 1 error
>> stops the
>> > > >> > other
>> > > >> > > > > > topics from being created
>> > > >> > > > > > - Why implement "partial blocking" instead of fully
>> async
>> > > or
>> > > >> > fully
>> > > >> > > > > > consistent?
>> > > >> > > > > > - See Cluster Consistent Blocking
>> > > >> > > > > > <
>> > > >> > > > >
>> > > >> > > >
>> > > >> >
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
>> > > >> > > > > >
>> > > >> > > > > > below
>> > > >> > > > > > - Why require the request to go to the controller?
>> > > >> > > > > > - The controller is responsible for the cluster
>> metadata
>> > > >> and
>> > > >> > its
>> > > >> > > > > > propagation
>> > > >> > > > > > - See Request Forwarding
>> > > >> > > > > > <
>> > > >> > > > >
>> > > >> > > >
>> > > >> >
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
>> > > >> > > > > >
>> > > >> > > > > > below
>> > > >> > > > > >
>> > > >> > > > > > Create Topics Response
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > > CreateTopics Response (Version: 0) => [topic_error_codes]
>> > > >> > > > > > topic_error_codes => topic error_code
>> > > >> > > > > > topic => STRING
>> > > >> > > > > > error_code => INT16
>> > > >> > > > > >
>> > > >> > > > > > CreateTopicsResponse contains a map between topic and
>> topic
>> > > >> > creation
>> > > >> > > > > > result error code (see New Protocol Errors
>> > > >> > > > > > <
>> > > >> > > > >
>> > > >> > > >
>> > > >> >
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
>> > > >> > > > > >
>> > > >> > > > > > ).
>> > > >> > > > > >
>> > > >> > > > >
>> > > >> > > > > The KIP is available here for reference (linked to the
>> Create
>> > > >> Topics
>> > > >> > > > schema
>> > > >> > > > > section):
>> > > >> > > > > *
>> > > >> > > > >
>> > > >> > > >
>> > > >> >
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
>> > > >> > > > > <
>> > > >> > > > >
>> > > >> > > >
>> > > >> >
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
>> > > >> > > > > >*
>> > > >> > > > >
>> > > >> > > > > A pull request is available implementing the proposed
>> changes
>> > > here:
>> > > >> > > > > https://github.com/apache/kafka/pull/1489
>> > > >> > > > >
>> > > >> > > > > Here is a link to the past discussion on the mailing list:
>> > > >> > > > > *
>> > > >> > > > >
>> > > >> > > >
>> > > >> >
>> > > >>
>> > >
>> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
>> > > >> > > > > <
>> > > >> > > > >
>> > > >> > > >
>> > > >> >
>> > > >>
>> > >
>> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
>> > > >> > > > > >*
>> > > >> > > > >
>> > > >> > > > > Thank you,
>> > > >> > > > > Grant
>> > > >> > > > > --
>> > > >> > > > > Grant Henke
>> > > >> > > > > Software Engineer | Cloudera
>> > > >> > > > > grant@cloudera.com | twitter.com/gchenke |
>> > > >> > linkedin.com/in/granthenke
>> > > >> > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> > >
>> > > >> > >
>> > > >> > > --
>> > > >> > > -- Guozhang
>> > > >> >
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Thanks,
>> > > >> Ewen
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Grant Henke
>> > > > Software Engineer | Cloudera
>> > > > grant@cloudera.com | twitter.com/gchenke |
>> linkedin.com/in/granthenke
>> > >
>>
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Tom Crayford <tc...@heroku.com>.
+1 (non-binding)
On Mon, Jun 20, 2016 at 8:07 PM, Harsha <ka...@harsha.io> wrote:
> +1 (binding)
> -Harsha
>
> On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote:
> > +1 (binding)
> >
> > On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers <da...@gmail.com>
> > wrote:
> >
> > > +1 -- thanks for the update
> > >
> > > On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke <gh...@cloudera.com>
> wrote:
> > > > I have update the patch and wiki based on the feedback in the
> discussion
> > > > thread. The only change is that instead of logging and disconnecting
> in
> > > the
> > > > case of invalid messages (duplicate topics or both arguments) we now
> > > return
> > > > and InvalidRequest error back to the client for that topic.
> > > >
> > > > I would like to restart the vote now including that change. If you
> have
> > > > already voted, please revote in this thread.
> > > >
> > > > Thank you,
> > > > Grant
> > > >
> > > > On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> > > ewen@confluent.io>
> > > > wrote:
> > > >
> > > >> Don't necessarily want to add noise here, but I'm -1 based on the
> > > >> disconnect part. See discussion in other thread. (I'm +1 otherwise,
> and
> > > >> happy to have my vote applied assuming we clean up that one issue.)
> > > >>
> > > >> -Ewen
> > > >>
> > > >> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io> wrote:
> > > >>
> > > >> > +1 (binding)
> > > >> > Thanks,
> > > >> > Harsha
> > > >> >
> > > >> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> > > >> > > +1.
> > > >> > >
> > > >> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <ismael@juma.me.uk
> >
> > > >> wrote:
> > > >> > >
> > > >> > > > +1 (binding)
> > > >> > > >
> > > >> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <
> > > ghenke@cloudera.com>
> > > >> > wrote:
> > > >> > > >
> > > >> > > > > I would like to initiate the voting process for the "KIP-4
> > > Create
> > > >> > Topics
> > > >> > > > > Schema changes". This is not a vote for all of KIP-4, but
> > > >> > specifically
> > > >> > > > for
> > > >> > > > > the create topics changes. I have included the exact changes
> > > below
> > > >> > for
> > > >> > > > > clarity:
> > > >> > > > > >
> > > >> > > > > > Create Topics Request (KAFKA-2945
> > > >> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> > > >> > > > > >
> > > >> > > > > > CreateTopics Request (Version: 0) =>
> [create_topic_requests]
> > > >> > timeout
> > > >> > > > > > create_topic_requests => topic num_partitions
> > > >> replication_factor
> > > >> > > > > [replica_assignment] [configs]
> > > >> > > > > > topic => STRING
> > > >> > > > > > num_partitions => INT32
> > > >> > > > > > replication_factor => INT16
> > > >> > > > > > replica_assignment => partition_id [replicas]
> > > >> > > > > > partition_id => INT32
> > > >> > > > > > replicas => INT32
> > > >> > > > > > configs => config_key config_value
> > > >> > > > > > config_key => STRING
> > > >> > > > > > config_value => STRING
> > > >> > > > > > timeout => INT32
> > > >> > > > > >
> > > >> > > > > > CreateTopicsRequest is a batch request to initiate topic
> > > creation
> > > >> > with
> > > >> > > > > > either predefined or automatic replica assignment and
> > > optionally
> > > >> > topic
> > > >> > > > > > configuration.
> > > >> > > > > >
> > > >> > > > > > Request semantics:
> > > >> > > > > >
> > > >> > > > > > 1. Must be sent to the controller broker
> > > >> > > > > > 2. If there are multiple instructions for the same
> topic in
> > > >> one
> > > >> > > > > > request an InvalidRequestException will be logged on
> the
> > > >> broker
> > > >> > and
> > > >> > > > > the
> > > >> > > > > > client will be disconnected.
> > > >> > > > > > - This is because the list of topics is modeled
> server
> > > side
> > > >> > as a
> > > >> > > > > > map with TopicName as the key
> > > >> > > > > > 3. The principal must be authorized to the "Create"
> > > Operation
> > > >> > on the
> > > >> > > > > > "Cluster" resource to create topics.
> > > >> > > > > > - Unauthorized requests will receive a
> > > >> > > > > ClusterAuthorizationException
> > > >> > > > > > 4.
> > > >> > > > > >
> > > >> > > > > > Only one from ReplicaAssignment or (num_partitions +
> > > >> > > > > replication_factor
> > > >> > > > > > ), can be defined in one instruction.
> > > >> > > > > > - If both parameters are specified an
> > > InvalidRequestException
> > > >> > will
> > > >> > > > be
> > > >> > > > > > logged on the broker and the client will be
> > > disconnected.
> > > >> > > > > > - In the case ReplicaAssignment is defined number of
> > > >> > partitions
> > > >> > > > and
> > > >> > > > > > replicas will be calculated from the supplied
> > > >> > replica_assignment.
> > > >> > > > > > - In the case of defined (num_partitions +
> > > >> > replication_factor)
> > > >> > > > > > replica assignment will be automatically generated
> by
> > > the
> > > >> > server.
> > > >> > > > > > - One or the other must be defined. The existing
> broker
> > > >> side
> > > >> > auto
> > > >> > > > > > create defaults will not be used
> > > >> > > > > > (default.replication.factor, num.partitions). The
> client
> > > >> > > > > implementation can
> > > >> > > > > > have defaults for these options when generating the
> > > >> messages.
> > > >> > > > > > - The first replica in [replicas] is assumed to be
> the
> > > >> > preferred
> > > >> > > > > > leader. This matches current behavior elsewhere.
> > > >> > > > > > 5. Setting a timeout > 0 will allow the request to
> block
> > > until
> > > >> > the
> > > >> > > > > > topic metadata is "complete" on the controller node.
> > > >> > > > > > - Complete means the local topic metadata cache been
> > > >> > completely
> > > >> > > > > > populated and all partitions have leaders
> > > >> > > > > > - The topic metadata is updated when the
> controller
> > > >> sends
> > > >> > out
> > > >> > > > > > update metadata requests to the brokers
> > > >> > > > > > - If a timeout error occurs, the topic could still
> be
> > > >> created
> > > >> > > > > > successfully at a later time. Its up to the client
> to
> > > query
> > > >> > for
> > > >> > > > > the state
> > > >> > > > > > at that point.
> > > >> > > > > > 6. Setting a timeout <= 0 will validate arguments and
> > > trigger
> > > >> > the
> > > >> > > > > > create topics and return immediately.
> > > >> > > > > > - This is essentially the fully asynchronous mode we
> > > have
> > > >> in
> > > >> > the
> > > >> > > > > > Zookeeper tools today.
> > > >> > > > > > - The error code in the response will either
> contain an
> > > >> > argument
> > > >> > > > > > validation exception or a timeout exception. If you
> > > >> receive a
> > > >> > > > > timeout
> > > >> > > > > > exception, because you asked for 0 timeout, you can
> > > assume
> > > >> > the
> > > >> > > > > message was
> > > >> > > > > > valid and the topic creation was triggered.
> > > >> > > > > > 7. The request is not transactional.
> > > >> > > > > > 1. If an error occurs on one topic, the others could
> > > still
> > > >> be
> > > >> > > > > > created.
> > > >> > > > > > 2. Errors are reported independently.
> > > >> > > > > >
> > > >> > > > > > QA:
> > > >> > > > > >
> > > >> > > > > > - Why is CreateTopicsRequest a batch request?
> > > >> > > > > > - Scenarios where tools or admins want to create
> many
> > > >> topics
> > > >> > > > should
> > > >> > > > > > be able to with fewer requests
> > > >> > > > > > - Example: MirrorMaker may want to create the topics
> > > >> > downstream
> > > >> > > > > > - What happens if some topics error immediately? Will
> it
> > > >> > > > > > return immediately?
> > > >> > > > > > - The request will block until all topics have
> either
> > > been
> > > >> > > > created,
> > > >> > > > > > errors, or the timeout has been hit
> > > >> > > > > > - There is no "short circuiting" where 1 error
> stops the
> > > >> > other
> > > >> > > > > > topics from being created
> > > >> > > > > > - Why implement "partial blocking" instead of fully
> async
> > > or
> > > >> > fully
> > > >> > > > > > consistent?
> > > >> > > > > > - See Cluster Consistent Blocking
> > > >> > > > > > <
> > > >> > > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> > > >> > > > > >
> > > >> > > > > > below
> > > >> > > > > > - Why require the request to go to the controller?
> > > >> > > > > > - The controller is responsible for the cluster
> metadata
> > > >> and
> > > >> > its
> > > >> > > > > > propagation
> > > >> > > > > > - See Request Forwarding
> > > >> > > > > > <
> > > >> > > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> > > >> > > > > >
> > > >> > > > > > below
> > > >> > > > > >
> > > >> > > > > > Create Topics Response
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > CreateTopics Response (Version: 0) => [topic_error_codes]
> > > >> > > > > > topic_error_codes => topic error_code
> > > >> > > > > > topic => STRING
> > > >> > > > > > error_code => INT16
> > > >> > > > > >
> > > >> > > > > > CreateTopicsResponse contains a map between topic and
> topic
> > > >> > creation
> > > >> > > > > > result error code (see New Protocol Errors
> > > >> > > > > > <
> > > >> > > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> > > >> > > > > >
> > > >> > > > > > ).
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > > The KIP is available here for reference (linked to the
> Create
> > > >> Topics
> > > >> > > > schema
> > > >> > > > > section):
> > > >> > > > > *
> > > >> > > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > >> > > > > <
> > > >> > > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > >> > > > > >*
> > > >> > > > >
> > > >> > > > > A pull request is available implementing the proposed
> changes
> > > here:
> > > >> > > > > https://github.com/apache/kafka/pull/1489
> > > >> > > > >
> > > >> > > > > Here is a link to the past discussion on the mailing list:
> > > >> > > > > *
> > > >> > > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > >> > > > > <
> > > >> > > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > >> > > > > >*
> > > >> > > > >
> > > >> > > > > Thank you,
> > > >> > > > > Grant
> > > >> > > > > --
> > > >> > > > > Grant Henke
> > > >> > > > > Software Engineer | Cloudera
> > > >> > > > > grant@cloudera.com | twitter.com/gchenke |
> > > >> > linkedin.com/in/granthenke
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > -- Guozhang
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Thanks,
> > > >> Ewen
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Grant Henke
> > > > Software Engineer | Cloudera
> > > > grant@cloudera.com | twitter.com/gchenke |
> linkedin.com/in/granthenke
> > >
>
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Harsha <ka...@harsha.io>.
+1 (binding)
-Harsha
On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote:
> +1 (binding)
>
> On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers <da...@gmail.com>
> wrote:
>
> > +1 -- thanks for the update
> >
> > On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke <gh...@cloudera.com> wrote:
> > > I have update the patch and wiki based on the feedback in the discussion
> > > thread. The only change is that instead of logging and disconnecting in
> > the
> > > case of invalid messages (duplicate topics or both arguments) we now
> > return
> > > and InvalidRequest error back to the client for that topic.
> > >
> > > I would like to restart the vote now including that change. If you have
> > > already voted, please revote in this thread.
> > >
> > > Thank you,
> > > Grant
> > >
> > > On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> > ewen@confluent.io>
> > > wrote:
> > >
> > >> Don't necessarily want to add noise here, but I'm -1 based on the
> > >> disconnect part. See discussion in other thread. (I'm +1 otherwise, and
> > >> happy to have my vote applied assuming we clean up that one issue.)
> > >>
> > >> -Ewen
> > >>
> > >> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io> wrote:
> > >>
> > >> > +1 (binding)
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> > >> > > +1.
> > >> > >
> > >> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <is...@juma.me.uk>
> > >> wrote:
> > >> > >
> > >> > > > +1 (binding)
> > >> > > >
> > >> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <
> > ghenke@cloudera.com>
> > >> > wrote:
> > >> > > >
> > >> > > > > I would like to initiate the voting process for the "KIP-4
> > Create
> > >> > Topics
> > >> > > > > Schema changes". This is not a vote for all of KIP-4, but
> > >> > specifically
> > >> > > > for
> > >> > > > > the create topics changes. I have included the exact changes
> > below
> > >> > for
> > >> > > > > clarity:
> > >> > > > > >
> > >> > > > > > Create Topics Request (KAFKA-2945
> > >> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> > >> > > > > >
> > >> > > > > > CreateTopics Request (Version: 0) => [create_topic_requests]
> > >> > timeout
> > >> > > > > > create_topic_requests => topic num_partitions
> > >> replication_factor
> > >> > > > > [replica_assignment] [configs]
> > >> > > > > > topic => STRING
> > >> > > > > > num_partitions => INT32
> > >> > > > > > replication_factor => INT16
> > >> > > > > > replica_assignment => partition_id [replicas]
> > >> > > > > > partition_id => INT32
> > >> > > > > > replicas => INT32
> > >> > > > > > configs => config_key config_value
> > >> > > > > > config_key => STRING
> > >> > > > > > config_value => STRING
> > >> > > > > > timeout => INT32
> > >> > > > > >
> > >> > > > > > CreateTopicsRequest is a batch request to initiate topic
> > creation
> > >> > with
> > >> > > > > > either predefined or automatic replica assignment and
> > optionally
> > >> > topic
> > >> > > > > > configuration.
> > >> > > > > >
> > >> > > > > > Request semantics:
> > >> > > > > >
> > >> > > > > > 1. Must be sent to the controller broker
> > >> > > > > > 2. If there are multiple instructions for the same topic in
> > >> one
> > >> > > > > > request an InvalidRequestException will be logged on the
> > >> broker
> > >> > and
> > >> > > > > the
> > >> > > > > > client will be disconnected.
> > >> > > > > > - This is because the list of topics is modeled server
> > side
> > >> > as a
> > >> > > > > > map with TopicName as the key
> > >> > > > > > 3. The principal must be authorized to the "Create"
> > Operation
> > >> > on the
> > >> > > > > > "Cluster" resource to create topics.
> > >> > > > > > - Unauthorized requests will receive a
> > >> > > > > ClusterAuthorizationException
> > >> > > > > > 4.
> > >> > > > > >
> > >> > > > > > Only one from ReplicaAssignment or (num_partitions +
> > >> > > > > replication_factor
> > >> > > > > > ), can be defined in one instruction.
> > >> > > > > > - If both parameters are specified an
> > InvalidRequestException
> > >> > will
> > >> > > > be
> > >> > > > > > logged on the broker and the client will be
> > disconnected.
> > >> > > > > > - In the case ReplicaAssignment is defined number of
> > >> > partitions
> > >> > > > and
> > >> > > > > > replicas will be calculated from the supplied
> > >> > replica_assignment.
> > >> > > > > > - In the case of defined (num_partitions +
> > >> > replication_factor)
> > >> > > > > > replica assignment will be automatically generated by
> > the
> > >> > server.
> > >> > > > > > - One or the other must be defined. The existing broker
> > >> side
> > >> > auto
> > >> > > > > > create defaults will not be used
> > >> > > > > > (default.replication.factor, num.partitions). The client
> > >> > > > > implementation can
> > >> > > > > > have defaults for these options when generating the
> > >> messages.
> > >> > > > > > - The first replica in [replicas] is assumed to be the
> > >> > preferred
> > >> > > > > > leader. This matches current behavior elsewhere.
> > >> > > > > > 5. Setting a timeout > 0 will allow the request to block
> > until
> > >> > the
> > >> > > > > > topic metadata is "complete" on the controller node.
> > >> > > > > > - Complete means the local topic metadata cache been
> > >> > completely
> > >> > > > > > populated and all partitions have leaders
> > >> > > > > > - The topic metadata is updated when the controller
> > >> sends
> > >> > out
> > >> > > > > > update metadata requests to the brokers
> > >> > > > > > - If a timeout error occurs, the topic could still be
> > >> created
> > >> > > > > > successfully at a later time. Its up to the client to
> > query
> > >> > for
> > >> > > > > the state
> > >> > > > > > at that point.
> > >> > > > > > 6. Setting a timeout <= 0 will validate arguments and
> > trigger
> > >> > the
> > >> > > > > > create topics and return immediately.
> > >> > > > > > - This is essentially the fully asynchronous mode we
> > have
> > >> in
> > >> > the
> > >> > > > > > Zookeeper tools today.
> > >> > > > > > - The error code in the response will either contain an
> > >> > argument
> > >> > > > > > validation exception or a timeout exception. If you
> > >> receive a
> > >> > > > > timeout
> > >> > > > > > exception, because you asked for 0 timeout, you can
> > assume
> > >> > the
> > >> > > > > message was
> > >> > > > > > valid and the topic creation was triggered.
> > >> > > > > > 7. The request is not transactional.
> > >> > > > > > 1. If an error occurs on one topic, the others could
> > still
> > >> be
> > >> > > > > > created.
> > >> > > > > > 2. Errors are reported independently.
> > >> > > > > >
> > >> > > > > > QA:
> > >> > > > > >
> > >> > > > > > - Why is CreateTopicsRequest a batch request?
> > >> > > > > > - Scenarios where tools or admins want to create many
> > >> topics
> > >> > > > should
> > >> > > > > > be able to with fewer requests
> > >> > > > > > - Example: MirrorMaker may want to create the topics
> > >> > downstream
> > >> > > > > > - What happens if some topics error immediately? Will it
> > >> > > > > > return immediately?
> > >> > > > > > - The request will block until all topics have either
> > been
> > >> > > > created,
> > >> > > > > > errors, or the timeout has been hit
> > >> > > > > > - There is no "short circuiting" where 1 error stops the
> > >> > other
> > >> > > > > > topics from being created
> > >> > > > > > - Why implement "partial blocking" instead of fully async
> > or
> > >> > fully
> > >> > > > > > consistent?
> > >> > > > > > - See Cluster Consistent Blocking
> > >> > > > > > <
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> > >> > > > > >
> > >> > > > > > below
> > >> > > > > > - Why require the request to go to the controller?
> > >> > > > > > - The controller is responsible for the cluster metadata
> > >> and
> > >> > its
> > >> > > > > > propagation
> > >> > > > > > - See Request Forwarding
> > >> > > > > > <
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> > >> > > > > >
> > >> > > > > > below
> > >> > > > > >
> > >> > > > > > Create Topics Response
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > CreateTopics Response (Version: 0) => [topic_error_codes]
> > >> > > > > > topic_error_codes => topic error_code
> > >> > > > > > topic => STRING
> > >> > > > > > error_code => INT16
> > >> > > > > >
> > >> > > > > > CreateTopicsResponse contains a map between topic and topic
> > >> > creation
> > >> > > > > > result error code (see New Protocol Errors
> > >> > > > > > <
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> > >> > > > > >
> > >> > > > > > ).
> > >> > > > > >
> > >> > > > >
> > >> > > > > The KIP is available here for reference (linked to the Create
> > >> Topics
> > >> > > > schema
> > >> > > > > section):
> > >> > > > > *
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > >> > > > > <
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > >> > > > > >*
> > >> > > > >
> > >> > > > > A pull request is available implementing the proposed changes
> > here:
> > >> > > > > https://github.com/apache/kafka/pull/1489
> > >> > > > >
> > >> > > > > Here is a link to the past discussion on the mailing list:
> > >> > > > > *
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > >> > > > > <
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > >> > > > > >*
> > >> > > > >
> > >> > > > > Thank you,
> > >> > > > > Grant
> > >> > > > > --
> > >> > > > > Grant Henke
> > >> > > > > Software Engineer | Cloudera
> > >> > > > > grant@cloudera.com | twitter.com/gchenke |
> > >> > linkedin.com/in/granthenke
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > -- Guozhang
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Ewen
> > >>
> > >
> > >
> > >
> > > --
> > > Grant Henke
> > > Software Engineer | Cloudera
> > > grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Ismael Juma <is...@juma.me.uk>.
+1 (binding)
On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers <da...@gmail.com> wrote:
> +1 -- thanks for the update
>
> On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke <gh...@cloudera.com> wrote:
> > I have update the patch and wiki based on the feedback in the discussion
> > thread. The only change is that instead of logging and disconnecting in
> the
> > case of invalid messages (duplicate topics or both arguments) we now
> return
> > and InvalidRequest error back to the client for that topic.
> >
> > I would like to restart the vote now including that change. If you have
> > already voted, please revote in this thread.
> >
> > Thank you,
> > Grant
> >
> > On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> ewen@confluent.io>
> > wrote:
> >
> >> Don't necessarily want to add noise here, but I'm -1 based on the
> >> disconnect part. See discussion in other thread. (I'm +1 otherwise, and
> >> happy to have my vote applied assuming we clean up that one issue.)
> >>
> >> -Ewen
> >>
> >> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io> wrote:
> >>
> >> > +1 (binding)
> >> > Thanks,
> >> > Harsha
> >> >
> >> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> >> > > +1.
> >> > >
> >> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <is...@juma.me.uk>
> >> wrote:
> >> > >
> >> > > > +1 (binding)
> >> > > >
> >> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <
> ghenke@cloudera.com>
> >> > wrote:
> >> > > >
> >> > > > > I would like to initiate the voting process for the "KIP-4
> Create
> >> > Topics
> >> > > > > Schema changes". This is not a vote for all of KIP-4, but
> >> > specifically
> >> > > > for
> >> > > > > the create topics changes. I have included the exact changes
> below
> >> > for
> >> > > > > clarity:
> >> > > > > >
> >> > > > > > Create Topics Request (KAFKA-2945
> >> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> >> > > > > >
> >> > > > > > CreateTopics Request (Version: 0) => [create_topic_requests]
> >> > timeout
> >> > > > > > create_topic_requests => topic num_partitions
> >> replication_factor
> >> > > > > [replica_assignment] [configs]
> >> > > > > > topic => STRING
> >> > > > > > num_partitions => INT32
> >> > > > > > replication_factor => INT16
> >> > > > > > replica_assignment => partition_id [replicas]
> >> > > > > > partition_id => INT32
> >> > > > > > replicas => INT32
> >> > > > > > configs => config_key config_value
> >> > > > > > config_key => STRING
> >> > > > > > config_value => STRING
> >> > > > > > timeout => INT32
> >> > > > > >
> >> > > > > > CreateTopicsRequest is a batch request to initiate topic
> creation
> >> > with
> >> > > > > > either predefined or automatic replica assignment and
> optionally
> >> > topic
> >> > > > > > configuration.
> >> > > > > >
> >> > > > > > Request semantics:
> >> > > > > >
> >> > > > > > 1. Must be sent to the controller broker
> >> > > > > > 2. If there are multiple instructions for the same topic in
> >> one
> >> > > > > > request an InvalidRequestException will be logged on the
> >> broker
> >> > and
> >> > > > > the
> >> > > > > > client will be disconnected.
> >> > > > > > - This is because the list of topics is modeled server
> side
> >> > as a
> >> > > > > > map with TopicName as the key
> >> > > > > > 3. The principal must be authorized to the "Create"
> Operation
> >> > on the
> >> > > > > > "Cluster" resource to create topics.
> >> > > > > > - Unauthorized requests will receive a
> >> > > > > ClusterAuthorizationException
> >> > > > > > 4.
> >> > > > > >
> >> > > > > > Only one from ReplicaAssignment or (num_partitions +
> >> > > > > replication_factor
> >> > > > > > ), can be defined in one instruction.
> >> > > > > > - If both parameters are specified an
> InvalidRequestException
> >> > will
> >> > > > be
> >> > > > > > logged on the broker and the client will be
> disconnected.
> >> > > > > > - In the case ReplicaAssignment is defined number of
> >> > partitions
> >> > > > and
> >> > > > > > replicas will be calculated from the supplied
> >> > replica_assignment.
> >> > > > > > - In the case of defined (num_partitions +
> >> > replication_factor)
> >> > > > > > replica assignment will be automatically generated by
> the
> >> > server.
> >> > > > > > - One or the other must be defined. The existing broker
> >> side
> >> > auto
> >> > > > > > create defaults will not be used
> >> > > > > > (default.replication.factor, num.partitions). The client
> >> > > > > implementation can
> >> > > > > > have defaults for these options when generating the
> >> messages.
> >> > > > > > - The first replica in [replicas] is assumed to be the
> >> > preferred
> >> > > > > > leader. This matches current behavior elsewhere.
> >> > > > > > 5. Setting a timeout > 0 will allow the request to block
> until
> >> > the
> >> > > > > > topic metadata is "complete" on the controller node.
> >> > > > > > - Complete means the local topic metadata cache been
> >> > completely
> >> > > > > > populated and all partitions have leaders
> >> > > > > > - The topic metadata is updated when the controller
> >> sends
> >> > out
> >> > > > > > update metadata requests to the brokers
> >> > > > > > - If a timeout error occurs, the topic could still be
> >> created
> >> > > > > > successfully at a later time. Its up to the client to
> query
> >> > for
> >> > > > > the state
> >> > > > > > at that point.
> >> > > > > > 6. Setting a timeout <= 0 will validate arguments and
> trigger
> >> > the
> >> > > > > > create topics and return immediately.
> >> > > > > > - This is essentially the fully asynchronous mode we
> have
> >> in
> >> > the
> >> > > > > > Zookeeper tools today.
> >> > > > > > - The error code in the response will either contain an
> >> > argument
> >> > > > > > validation exception or a timeout exception. If you
> >> receive a
> >> > > > > timeout
> >> > > > > > exception, because you asked for 0 timeout, you can
> assume
> >> > the
> >> > > > > message was
> >> > > > > > valid and the topic creation was triggered.
> >> > > > > > 7. The request is not transactional.
> >> > > > > > 1. If an error occurs on one topic, the others could
> still
> >> be
> >> > > > > > created.
> >> > > > > > 2. Errors are reported independently.
> >> > > > > >
> >> > > > > > QA:
> >> > > > > >
> >> > > > > > - Why is CreateTopicsRequest a batch request?
> >> > > > > > - Scenarios where tools or admins want to create many
> >> topics
> >> > > > should
> >> > > > > > be able to with fewer requests
> >> > > > > > - Example: MirrorMaker may want to create the topics
> >> > downstream
> >> > > > > > - What happens if some topics error immediately? Will it
> >> > > > > > return immediately?
> >> > > > > > - The request will block until all topics have either
> been
> >> > > > created,
> >> > > > > > errors, or the timeout has been hit
> >> > > > > > - There is no "short circuiting" where 1 error stops the
> >> > other
> >> > > > > > topics from being created
> >> > > > > > - Why implement "partial blocking" instead of fully async
> or
> >> > fully
> >> > > > > > consistent?
> >> > > > > > - See Cluster Consistent Blocking
> >> > > > > > <
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> >> > > > > >
> >> > > > > > below
> >> > > > > > - Why require the request to go to the controller?
> >> > > > > > - The controller is responsible for the cluster metadata
> >> and
> >> > its
> >> > > > > > propagation
> >> > > > > > - See Request Forwarding
> >> > > > > > <
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> >> > > > > >
> >> > > > > > below
> >> > > > > >
> >> > > > > > Create Topics Response
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > CreateTopics Response (Version: 0) => [topic_error_codes]
> >> > > > > > topic_error_codes => topic error_code
> >> > > > > > topic => STRING
> >> > > > > > error_code => INT16
> >> > > > > >
> >> > > > > > CreateTopicsResponse contains a map between topic and topic
> >> > creation
> >> > > > > > result error code (see New Protocol Errors
> >> > > > > > <
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> >> > > > > >
> >> > > > > > ).
> >> > > > > >
> >> > > > >
> >> > > > > The KIP is available here for reference (linked to the Create
> >> Topics
> >> > > > schema
> >> > > > > section):
> >> > > > > *
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> >> > > > > <
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> >> > > > > >*
> >> > > > >
> >> > > > > A pull request is available implementing the proposed changes
> here:
> >> > > > > https://github.com/apache/kafka/pull/1489
> >> > > > >
> >> > > > > Here is a link to the past discussion on the mailing list:
> >> > > > > *
> >> > > > >
> >> > > >
> >> >
> >>
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> >> > > > > <
> >> > > > >
> >> > > >
> >> >
> >>
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> >> > > > > >*
> >> > > > >
> >> > > > > Thank you,
> >> > > > > Grant
> >> > > > > --
> >> > > > > Grant Henke
> >> > > > > Software Engineer | Cloudera
> >> > > > > grant@cloudera.com | twitter.com/gchenke |
> >> > linkedin.com/in/granthenke
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > -- Guozhang
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Ewen
> >>
> >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Dana Powers <da...@gmail.com>.
+1 -- thanks for the update
On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke <gh...@cloudera.com> wrote:
> I have update the patch and wiki based on the feedback in the discussion
> thread. The only change is that instead of logging and disconnecting in the
> case of invalid messages (duplicate topics or both arguments) we now return
> and InvalidRequest error back to the client for that topic.
>
> I would like to restart the vote now including that change. If you have
> already voted, please revote in this thread.
>
> Thank you,
> Grant
>
> On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
>> Don't necessarily want to add noise here, but I'm -1 based on the
>> disconnect part. See discussion in other thread. (I'm +1 otherwise, and
>> happy to have my vote applied assuming we clean up that one issue.)
>>
>> -Ewen
>>
>> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io> wrote:
>>
>> > +1 (binding)
>> > Thanks,
>> > Harsha
>> >
>> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
>> > > +1.
>> > >
>> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <is...@juma.me.uk>
>> wrote:
>> > >
>> > > > +1 (binding)
>> > > >
>> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <gh...@cloudera.com>
>> > wrote:
>> > > >
>> > > > > I would like to initiate the voting process for the "KIP-4 Create
>> > Topics
>> > > > > Schema changes". This is not a vote for all of KIP-4, but
>> > specifically
>> > > > for
>> > > > > the create topics changes. I have included the exact changes below
>> > for
>> > > > > clarity:
>> > > > > >
>> > > > > > Create Topics Request (KAFKA-2945
>> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
>> > > > > >
>> > > > > > CreateTopics Request (Version: 0) => [create_topic_requests]
>> > timeout
>> > > > > > create_topic_requests => topic num_partitions
>> replication_factor
>> > > > > [replica_assignment] [configs]
>> > > > > > topic => STRING
>> > > > > > num_partitions => INT32
>> > > > > > replication_factor => INT16
>> > > > > > replica_assignment => partition_id [replicas]
>> > > > > > partition_id => INT32
>> > > > > > replicas => INT32
>> > > > > > configs => config_key config_value
>> > > > > > config_key => STRING
>> > > > > > config_value => STRING
>> > > > > > timeout => INT32
>> > > > > >
>> > > > > > CreateTopicsRequest is a batch request to initiate topic creation
>> > with
>> > > > > > either predefined or automatic replica assignment and optionally
>> > topic
>> > > > > > configuration.
>> > > > > >
>> > > > > > Request semantics:
>> > > > > >
>> > > > > > 1. Must be sent to the controller broker
>> > > > > > 2. If there are multiple instructions for the same topic in
>> one
>> > > > > > request an InvalidRequestException will be logged on the
>> broker
>> > and
>> > > > > the
>> > > > > > client will be disconnected.
>> > > > > > - This is because the list of topics is modeled server side
>> > as a
>> > > > > > map with TopicName as the key
>> > > > > > 3. The principal must be authorized to the "Create" Operation
>> > on the
>> > > > > > "Cluster" resource to create topics.
>> > > > > > - Unauthorized requests will receive a
>> > > > > ClusterAuthorizationException
>> > > > > > 4.
>> > > > > >
>> > > > > > Only one from ReplicaAssignment or (num_partitions +
>> > > > > replication_factor
>> > > > > > ), can be defined in one instruction.
>> > > > > > - If both parameters are specified an InvalidRequestException
>> > will
>> > > > be
>> > > > > > logged on the broker and the client will be disconnected.
>> > > > > > - In the case ReplicaAssignment is defined number of
>> > partitions
>> > > > and
>> > > > > > replicas will be calculated from the supplied
>> > replica_assignment.
>> > > > > > - In the case of defined (num_partitions +
>> > replication_factor)
>> > > > > > replica assignment will be automatically generated by the
>> > server.
>> > > > > > - One or the other must be defined. The existing broker
>> side
>> > auto
>> > > > > > create defaults will not be used
>> > > > > > (default.replication.factor, num.partitions). The client
>> > > > > implementation can
>> > > > > > have defaults for these options when generating the
>> messages.
>> > > > > > - The first replica in [replicas] is assumed to be the
>> > preferred
>> > > > > > leader. This matches current behavior elsewhere.
>> > > > > > 5. Setting a timeout > 0 will allow the request to block until
>> > the
>> > > > > > topic metadata is "complete" on the controller node.
>> > > > > > - Complete means the local topic metadata cache been
>> > completely
>> > > > > > populated and all partitions have leaders
>> > > > > > - The topic metadata is updated when the controller
>> sends
>> > out
>> > > > > > update metadata requests to the brokers
>> > > > > > - If a timeout error occurs, the topic could still be
>> created
>> > > > > > successfully at a later time. Its up to the client to query
>> > for
>> > > > > the state
>> > > > > > at that point.
>> > > > > > 6. Setting a timeout <= 0 will validate arguments and trigger
>> > the
>> > > > > > create topics and return immediately.
>> > > > > > - This is essentially the fully asynchronous mode we have
>> in
>> > the
>> > > > > > Zookeeper tools today.
>> > > > > > - The error code in the response will either contain an
>> > argument
>> > > > > > validation exception or a timeout exception. If you
>> receive a
>> > > > > timeout
>> > > > > > exception, because you asked for 0 timeout, you can assume
>> > the
>> > > > > message was
>> > > > > > valid and the topic creation was triggered.
>> > > > > > 7. The request is not transactional.
>> > > > > > 1. If an error occurs on one topic, the others could still
>> be
>> > > > > > created.
>> > > > > > 2. Errors are reported independently.
>> > > > > >
>> > > > > > QA:
>> > > > > >
>> > > > > > - Why is CreateTopicsRequest a batch request?
>> > > > > > - Scenarios where tools or admins want to create many
>> topics
>> > > > should
>> > > > > > be able to with fewer requests
>> > > > > > - Example: MirrorMaker may want to create the topics
>> > downstream
>> > > > > > - What happens if some topics error immediately? Will it
>> > > > > > return immediately?
>> > > > > > - The request will block until all topics have either been
>> > > > created,
>> > > > > > errors, or the timeout has been hit
>> > > > > > - There is no "short circuiting" where 1 error stops the
>> > other
>> > > > > > topics from being created
>> > > > > > - Why implement "partial blocking" instead of fully async or
>> > fully
>> > > > > > consistent?
>> > > > > > - See Cluster Consistent Blocking
>> > > > > > <
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
>> > > > > >
>> > > > > > below
>> > > > > > - Why require the request to go to the controller?
>> > > > > > - The controller is responsible for the cluster metadata
>> and
>> > its
>> > > > > > propagation
>> > > > > > - See Request Forwarding
>> > > > > > <
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
>> > > > > >
>> > > > > > below
>> > > > > >
>> > > > > > Create Topics Response
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > CreateTopics Response (Version: 0) => [topic_error_codes]
>> > > > > > topic_error_codes => topic error_code
>> > > > > > topic => STRING
>> > > > > > error_code => INT16
>> > > > > >
>> > > > > > CreateTopicsResponse contains a map between topic and topic
>> > creation
>> > > > > > result error code (see New Protocol Errors
>> > > > > > <
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
>> > > > > >
>> > > > > > ).
>> > > > > >
>> > > > >
>> > > > > The KIP is available here for reference (linked to the Create
>> Topics
>> > > > schema
>> > > > > section):
>> > > > > *
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
>> > > > > <
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
>> > > > > >*
>> > > > >
>> > > > > A pull request is available implementing the proposed changes here:
>> > > > > https://github.com/apache/kafka/pull/1489
>> > > > >
>> > > > > Here is a link to the past discussion on the mailing list:
>> > > > > *
>> > > > >
>> > > >
>> >
>> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
>> > > > > <
>> > > > >
>> > > >
>> >
>> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
>> > > > > >*
>> > > > >
>> > > > > Thank you,
>> > > > > Grant
>> > > > > --
>> > > > > Grant Henke
>> > > > > Software Engineer | Cloudera
>> > > > > grant@cloudera.com | twitter.com/gchenke |
>> > linkedin.com/in/granthenke
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -- Guozhang
>> >
>>
>>
>>
>> --
>> Thanks,
>> Ewen
>>
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Grant Henke <gh...@cloudera.com>.
I have update the patch and wiki based on the feedback in the discussion
thread. The only change is that instead of logging and disconnecting in the
case of invalid messages (duplicate topics or both arguments) we now return
and InvalidRequest error back to the client for that topic.
I would like to restart the vote now including that change. If you have
already voted, please revote in this thread.
Thank you,
Grant
On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Don't necessarily want to add noise here, but I'm -1 based on the
> disconnect part. See discussion in other thread. (I'm +1 otherwise, and
> happy to have my vote applied assuming we clean up that one issue.)
>
> -Ewen
>
> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io> wrote:
>
> > +1 (binding)
> > Thanks,
> > Harsha
> >
> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> > > +1.
> > >
> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <gh...@cloudera.com>
> > wrote:
> > > >
> > > > > I would like to initiate the voting process for the "KIP-4 Create
> > Topics
> > > > > Schema changes". This is not a vote for all of KIP-4, but
> > specifically
> > > > for
> > > > > the create topics changes. I have included the exact changes below
> > for
> > > > > clarity:
> > > > > >
> > > > > > Create Topics Request (KAFKA-2945
> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> > > > > >
> > > > > > CreateTopics Request (Version: 0) => [create_topic_requests]
> > timeout
> > > > > > create_topic_requests => topic num_partitions
> replication_factor
> > > > > [replica_assignment] [configs]
> > > > > > topic => STRING
> > > > > > num_partitions => INT32
> > > > > > replication_factor => INT16
> > > > > > replica_assignment => partition_id [replicas]
> > > > > > partition_id => INT32
> > > > > > replicas => INT32
> > > > > > configs => config_key config_value
> > > > > > config_key => STRING
> > > > > > config_value => STRING
> > > > > > timeout => INT32
> > > > > >
> > > > > > CreateTopicsRequest is a batch request to initiate topic creation
> > with
> > > > > > either predefined or automatic replica assignment and optionally
> > topic
> > > > > > configuration.
> > > > > >
> > > > > > Request semantics:
> > > > > >
> > > > > > 1. Must be sent to the controller broker
> > > > > > 2. If there are multiple instructions for the same topic in
> one
> > > > > > request an InvalidRequestException will be logged on the
> broker
> > and
> > > > > the
> > > > > > client will be disconnected.
> > > > > > - This is because the list of topics is modeled server side
> > as a
> > > > > > map with TopicName as the key
> > > > > > 3. The principal must be authorized to the "Create" Operation
> > on the
> > > > > > "Cluster" resource to create topics.
> > > > > > - Unauthorized requests will receive a
> > > > > ClusterAuthorizationException
> > > > > > 4.
> > > > > >
> > > > > > Only one from ReplicaAssignment or (num_partitions +
> > > > > replication_factor
> > > > > > ), can be defined in one instruction.
> > > > > > - If both parameters are specified an InvalidRequestException
> > will
> > > > be
> > > > > > logged on the broker and the client will be disconnected.
> > > > > > - In the case ReplicaAssignment is defined number of
> > partitions
> > > > and
> > > > > > replicas will be calculated from the supplied
> > replica_assignment.
> > > > > > - In the case of defined (num_partitions +
> > replication_factor)
> > > > > > replica assignment will be automatically generated by the
> > server.
> > > > > > - One or the other must be defined. The existing broker
> side
> > auto
> > > > > > create defaults will not be used
> > > > > > (default.replication.factor, num.partitions). The client
> > > > > implementation can
> > > > > > have defaults for these options when generating the
> messages.
> > > > > > - The first replica in [replicas] is assumed to be the
> > preferred
> > > > > > leader. This matches current behavior elsewhere.
> > > > > > 5. Setting a timeout > 0 will allow the request to block until
> > the
> > > > > > topic metadata is "complete" on the controller node.
> > > > > > - Complete means the local topic metadata cache been
> > completely
> > > > > > populated and all partitions have leaders
> > > > > > - The topic metadata is updated when the controller
> sends
> > out
> > > > > > update metadata requests to the brokers
> > > > > > - If a timeout error occurs, the topic could still be
> created
> > > > > > successfully at a later time. Its up to the client to query
> > for
> > > > > the state
> > > > > > at that point.
> > > > > > 6. Setting a timeout <= 0 will validate arguments and trigger
> > the
> > > > > > create topics and return immediately.
> > > > > > - This is essentially the fully asynchronous mode we have
> in
> > the
> > > > > > Zookeeper tools today.
> > > > > > - The error code in the response will either contain an
> > argument
> > > > > > validation exception or a timeout exception. If you
> receive a
> > > > > timeout
> > > > > > exception, because you asked for 0 timeout, you can assume
> > the
> > > > > message was
> > > > > > valid and the topic creation was triggered.
> > > > > > 7. The request is not transactional.
> > > > > > 1. If an error occurs on one topic, the others could still
> be
> > > > > > created.
> > > > > > 2. Errors are reported independently.
> > > > > >
> > > > > > QA:
> > > > > >
> > > > > > - Why is CreateTopicsRequest a batch request?
> > > > > > - Scenarios where tools or admins want to create many
> topics
> > > > should
> > > > > > be able to with fewer requests
> > > > > > - Example: MirrorMaker may want to create the topics
> > downstream
> > > > > > - What happens if some topics error immediately? Will it
> > > > > > return immediately?
> > > > > > - The request will block until all topics have either been
> > > > created,
> > > > > > errors, or the timeout has been hit
> > > > > > - There is no "short circuiting" where 1 error stops the
> > other
> > > > > > topics from being created
> > > > > > - Why implement "partial blocking" instead of fully async or
> > fully
> > > > > > consistent?
> > > > > > - See Cluster Consistent Blocking
> > > > > > <
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> > > > > >
> > > > > > below
> > > > > > - Why require the request to go to the controller?
> > > > > > - The controller is responsible for the cluster metadata
> and
> > its
> > > > > > propagation
> > > > > > - See Request Forwarding
> > > > > > <
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> > > > > >
> > > > > > below
> > > > > >
> > > > > > Create Topics Response
> > > > > >
> > > > > >
> > > > > >
> > > > > > CreateTopics Response (Version: 0) => [topic_error_codes]
> > > > > > topic_error_codes => topic error_code
> > > > > > topic => STRING
> > > > > > error_code => INT16
> > > > > >
> > > > > > CreateTopicsResponse contains a map between topic and topic
> > creation
> > > > > > result error code (see New Protocol Errors
> > > > > > <
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> > > > > >
> > > > > > ).
> > > > > >
> > > > >
> > > > > The KIP is available here for reference (linked to the Create
> Topics
> > > > schema
> > > > > section):
> > > > > *
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > > > <
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > > > >*
> > > > >
> > > > > A pull request is available implementing the proposed changes here:
> > > > > https://github.com/apache/kafka/pull/1489
> > > > >
> > > > > Here is a link to the past discussion on the mailing list:
> > > > > *
> > > > >
> > > >
> >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > > > <
> > > > >
> > > >
> >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > > > >*
> > > > >
> > > > > Thank you,
> > > > > Grant
> > > > > --
> > > > > Grant Henke
> > > > > Software Engineer | Cloudera
> > > > > grant@cloudera.com | twitter.com/gchenke |
> > linkedin.com/in/granthenke
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Don't necessarily want to add noise here, but I'm -1 based on the
disconnect part. See discussion in other thread. (I'm +1 otherwise, and
happy to have my vote applied assuming we clean up that one issue.)
-Ewen
On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io> wrote:
> +1 (binding)
> Thanks,
> Harsha
>
> On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> > +1.
> >
> > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > +1 (binding)
> > >
> > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <gh...@cloudera.com>
> wrote:
> > >
> > > > I would like to initiate the voting process for the "KIP-4 Create
> Topics
> > > > Schema changes". This is not a vote for all of KIP-4, but
> specifically
> > > for
> > > > the create topics changes. I have included the exact changes below
> for
> > > > clarity:
> > > > >
> > > > > Create Topics Request (KAFKA-2945
> > > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> > > > >
> > > > > CreateTopics Request (Version: 0) => [create_topic_requests]
> timeout
> > > > > create_topic_requests => topic num_partitions replication_factor
> > > > [replica_assignment] [configs]
> > > > > topic => STRING
> > > > > num_partitions => INT32
> > > > > replication_factor => INT16
> > > > > replica_assignment => partition_id [replicas]
> > > > > partition_id => INT32
> > > > > replicas => INT32
> > > > > configs => config_key config_value
> > > > > config_key => STRING
> > > > > config_value => STRING
> > > > > timeout => INT32
> > > > >
> > > > > CreateTopicsRequest is a batch request to initiate topic creation
> with
> > > > > either predefined or automatic replica assignment and optionally
> topic
> > > > > configuration.
> > > > >
> > > > > Request semantics:
> > > > >
> > > > > 1. Must be sent to the controller broker
> > > > > 2. If there are multiple instructions for the same topic in one
> > > > > request an InvalidRequestException will be logged on the broker
> and
> > > > the
> > > > > client will be disconnected.
> > > > > - This is because the list of topics is modeled server side
> as a
> > > > > map with TopicName as the key
> > > > > 3. The principal must be authorized to the "Create" Operation
> on the
> > > > > "Cluster" resource to create topics.
> > > > > - Unauthorized requests will receive a
> > > > ClusterAuthorizationException
> > > > > 4.
> > > > >
> > > > > Only one from ReplicaAssignment or (num_partitions +
> > > > replication_factor
> > > > > ), can be defined in one instruction.
> > > > > - If both parameters are specified an InvalidRequestException
> will
> > > be
> > > > > logged on the broker and the client will be disconnected.
> > > > > - In the case ReplicaAssignment is defined number of
> partitions
> > > and
> > > > > replicas will be calculated from the supplied
> replica_assignment.
> > > > > - In the case of defined (num_partitions +
> replication_factor)
> > > > > replica assignment will be automatically generated by the
> server.
> > > > > - One or the other must be defined. The existing broker side
> auto
> > > > > create defaults will not be used
> > > > > (default.replication.factor, num.partitions). The client
> > > > implementation can
> > > > > have defaults for these options when generating the messages.
> > > > > - The first replica in [replicas] is assumed to be the
> preferred
> > > > > leader. This matches current behavior elsewhere.
> > > > > 5. Setting a timeout > 0 will allow the request to block until
> the
> > > > > topic metadata is "complete" on the controller node.
> > > > > - Complete means the local topic metadata cache been
> completely
> > > > > populated and all partitions have leaders
> > > > > - The topic metadata is updated when the controller sends
> out
> > > > > update metadata requests to the brokers
> > > > > - If a timeout error occurs, the topic could still be created
> > > > > successfully at a later time. Its up to the client to query
> for
> > > > the state
> > > > > at that point.
> > > > > 6. Setting a timeout <= 0 will validate arguments and trigger
> the
> > > > > create topics and return immediately.
> > > > > - This is essentially the fully asynchronous mode we have in
> the
> > > > > Zookeeper tools today.
> > > > > - The error code in the response will either contain an
> argument
> > > > > validation exception or a timeout exception. If you receive a
> > > > timeout
> > > > > exception, because you asked for 0 timeout, you can assume
> the
> > > > message was
> > > > > valid and the topic creation was triggered.
> > > > > 7. The request is not transactional.
> > > > > 1. If an error occurs on one topic, the others could still be
> > > > > created.
> > > > > 2. Errors are reported independently.
> > > > >
> > > > > QA:
> > > > >
> > > > > - Why is CreateTopicsRequest a batch request?
> > > > > - Scenarios where tools or admins want to create many topics
> > > should
> > > > > be able to with fewer requests
> > > > > - Example: MirrorMaker may want to create the topics
> downstream
> > > > > - What happens if some topics error immediately? Will it
> > > > > return immediately?
> > > > > - The request will block until all topics have either been
> > > created,
> > > > > errors, or the timeout has been hit
> > > > > - There is no "short circuiting" where 1 error stops the
> other
> > > > > topics from being created
> > > > > - Why implement "partial blocking" instead of fully async or
> fully
> > > > > consistent?
> > > > > - See Cluster Consistent Blocking
> > > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> > > > >
> > > > > below
> > > > > - Why require the request to go to the controller?
> > > > > - The controller is responsible for the cluster metadata and
> its
> > > > > propagation
> > > > > - See Request Forwarding
> > > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> > > > >
> > > > > below
> > > > >
> > > > > Create Topics Response
> > > > >
> > > > >
> > > > >
> > > > > CreateTopics Response (Version: 0) => [topic_error_codes]
> > > > > topic_error_codes => topic error_code
> > > > > topic => STRING
> > > > > error_code => INT16
> > > > >
> > > > > CreateTopicsResponse contains a map between topic and topic
> creation
> > > > > result error code (see New Protocol Errors
> > > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> > > > >
> > > > > ).
> > > > >
> > > >
> > > > The KIP is available here for reference (linked to the Create Topics
> > > schema
> > > > section):
> > > > *
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > > >*
> > > >
> > > > A pull request is available implementing the proposed changes here:
> > > > https://github.com/apache/kafka/pull/1489
> > > >
> > > > Here is a link to the past discussion on the mailing list:
> > > > *
> > > >
> > >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > > <
> > > >
> > >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > > >*
> > > >
> > > > Thank you,
> > > > Grant
> > > > --
> > > > Grant Henke
> > > > Software Engineer | Cloudera
> > > > grant@cloudera.com | twitter.com/gchenke |
> linkedin.com/in/granthenke
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
>
--
Thanks,
Ewen
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Harsha <ka...@harsha.io>.
+1 (binding)
Thanks,
Harsha
On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> +1.
>
> On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > +1 (binding)
> >
> > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <gh...@cloudera.com> wrote:
> >
> > > I would like to initiate the voting process for the "KIP-4 Create Topics
> > > Schema changes". This is not a vote for all of KIP-4, but specifically
> > for
> > > the create topics changes. I have included the exact changes below for
> > > clarity:
> > > >
> > > > Create Topics Request (KAFKA-2945
> > > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> > > >
> > > > CreateTopics Request (Version: 0) => [create_topic_requests] timeout
> > > > create_topic_requests => topic num_partitions replication_factor
> > > [replica_assignment] [configs]
> > > > topic => STRING
> > > > num_partitions => INT32
> > > > replication_factor => INT16
> > > > replica_assignment => partition_id [replicas]
> > > > partition_id => INT32
> > > > replicas => INT32
> > > > configs => config_key config_value
> > > > config_key => STRING
> > > > config_value => STRING
> > > > timeout => INT32
> > > >
> > > > CreateTopicsRequest is a batch request to initiate topic creation with
> > > > either predefined or automatic replica assignment and optionally topic
> > > > configuration.
> > > >
> > > > Request semantics:
> > > >
> > > > 1. Must be sent to the controller broker
> > > > 2. If there are multiple instructions for the same topic in one
> > > > request an InvalidRequestException will be logged on the broker and
> > > the
> > > > client will be disconnected.
> > > > - This is because the list of topics is modeled server side as a
> > > > map with TopicName as the key
> > > > 3. The principal must be authorized to the "Create" Operation on the
> > > > "Cluster" resource to create topics.
> > > > - Unauthorized requests will receive a
> > > ClusterAuthorizationException
> > > > 4.
> > > >
> > > > Only one from ReplicaAssignment or (num_partitions +
> > > replication_factor
> > > > ), can be defined in one instruction.
> > > > - If both parameters are specified an InvalidRequestException will
> > be
> > > > logged on the broker and the client will be disconnected.
> > > > - In the case ReplicaAssignment is defined number of partitions
> > and
> > > > replicas will be calculated from the supplied replica_assignment.
> > > > - In the case of defined (num_partitions + replication_factor)
> > > > replica assignment will be automatically generated by the server.
> > > > - One or the other must be defined. The existing broker side auto
> > > > create defaults will not be used
> > > > (default.replication.factor, num.partitions). The client
> > > implementation can
> > > > have defaults for these options when generating the messages.
> > > > - The first replica in [replicas] is assumed to be the preferred
> > > > leader. This matches current behavior elsewhere.
> > > > 5. Setting a timeout > 0 will allow the request to block until the
> > > > topic metadata is "complete" on the controller node.
> > > > - Complete means the local topic metadata cache been completely
> > > > populated and all partitions have leaders
> > > > - The topic metadata is updated when the controller sends out
> > > > update metadata requests to the brokers
> > > > - If a timeout error occurs, the topic could still be created
> > > > successfully at a later time. Its up to the client to query for
> > > the state
> > > > at that point.
> > > > 6. Setting a timeout <= 0 will validate arguments and trigger the
> > > > create topics and return immediately.
> > > > - This is essentially the fully asynchronous mode we have in the
> > > > Zookeeper tools today.
> > > > - The error code in the response will either contain an argument
> > > > validation exception or a timeout exception. If you receive a
> > > timeout
> > > > exception, because you asked for 0 timeout, you can assume the
> > > message was
> > > > valid and the topic creation was triggered.
> > > > 7. The request is not transactional.
> > > > 1. If an error occurs on one topic, the others could still be
> > > > created.
> > > > 2. Errors are reported independently.
> > > >
> > > > QA:
> > > >
> > > > - Why is CreateTopicsRequest a batch request?
> > > > - Scenarios where tools or admins want to create many topics
> > should
> > > > be able to with fewer requests
> > > > - Example: MirrorMaker may want to create the topics downstream
> > > > - What happens if some topics error immediately? Will it
> > > > return immediately?
> > > > - The request will block until all topics have either been
> > created,
> > > > errors, or the timeout has been hit
> > > > - There is no "short circuiting" where 1 error stops the other
> > > > topics from being created
> > > > - Why implement "partial blocking" instead of fully async or fully
> > > > consistent?
> > > > - See Cluster Consistent Blocking
> > > > <
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> > > >
> > > > below
> > > > - Why require the request to go to the controller?
> > > > - The controller is responsible for the cluster metadata and its
> > > > propagation
> > > > - See Request Forwarding
> > > > <
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> > > >
> > > > below
> > > >
> > > > Create Topics Response
> > > >
> > > >
> > > >
> > > > CreateTopics Response (Version: 0) => [topic_error_codes]
> > > > topic_error_codes => topic error_code
> > > > topic => STRING
> > > > error_code => INT16
> > > >
> > > > CreateTopicsResponse contains a map between topic and topic creation
> > > > result error code (see New Protocol Errors
> > > > <
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> > > >
> > > > ).
> > > >
> > >
> > > The KIP is available here for reference (linked to the Create Topics
> > schema
> > > section):
> > > *
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > <
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > > >*
> > >
> > > A pull request is available implementing the proposed changes here:
> > > https://github.com/apache/kafka/pull/1489
> > >
> > > Here is a link to the past discussion on the mailing list:
> > > *
> > >
> > http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > <
> > >
> > http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > > >*
> > >
> > > Thank you,
> > > Grant
> > > --
> > > Grant Henke
> > > Software Engineer | Cloudera
> > > grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> > >
> >
>
>
>
> --
> -- Guozhang
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Guozhang Wang <wa...@gmail.com>.
+1.
On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <is...@juma.me.uk> wrote:
> +1 (binding)
>
> On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <gh...@cloudera.com> wrote:
>
> > I would like to initiate the voting process for the "KIP-4 Create Topics
> > Schema changes". This is not a vote for all of KIP-4, but specifically
> for
> > the create topics changes. I have included the exact changes below for
> > clarity:
> > >
> > > Create Topics Request (KAFKA-2945
> > > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> > >
> > > CreateTopics Request (Version: 0) => [create_topic_requests] timeout
> > > create_topic_requests => topic num_partitions replication_factor
> > [replica_assignment] [configs]
> > > topic => STRING
> > > num_partitions => INT32
> > > replication_factor => INT16
> > > replica_assignment => partition_id [replicas]
> > > partition_id => INT32
> > > replicas => INT32
> > > configs => config_key config_value
> > > config_key => STRING
> > > config_value => STRING
> > > timeout => INT32
> > >
> > > CreateTopicsRequest is a batch request to initiate topic creation with
> > > either predefined or automatic replica assignment and optionally topic
> > > configuration.
> > >
> > > Request semantics:
> > >
> > > 1. Must be sent to the controller broker
> > > 2. If there are multiple instructions for the same topic in one
> > > request an InvalidRequestException will be logged on the broker and
> > the
> > > client will be disconnected.
> > > - This is because the list of topics is modeled server side as a
> > > map with TopicName as the key
> > > 3. The principal must be authorized to the "Create" Operation on the
> > > "Cluster" resource to create topics.
> > > - Unauthorized requests will receive a
> > ClusterAuthorizationException
> > > 4.
> > >
> > > Only one from ReplicaAssignment or (num_partitions +
> > replication_factor
> > > ), can be defined in one instruction.
> > > - If both parameters are specified an InvalidRequestException will
> be
> > > logged on the broker and the client will be disconnected.
> > > - In the case ReplicaAssignment is defined number of partitions
> and
> > > replicas will be calculated from the supplied replica_assignment.
> > > - In the case of defined (num_partitions + replication_factor)
> > > replica assignment will be automatically generated by the server.
> > > - One or the other must be defined. The existing broker side auto
> > > create defaults will not be used
> > > (default.replication.factor, num.partitions). The client
> > implementation can
> > > have defaults for these options when generating the messages.
> > > - The first replica in [replicas] is assumed to be the preferred
> > > leader. This matches current behavior elsewhere.
> > > 5. Setting a timeout > 0 will allow the request to block until the
> > > topic metadata is "complete" on the controller node.
> > > - Complete means the local topic metadata cache been completely
> > > populated and all partitions have leaders
> > > - The topic metadata is updated when the controller sends out
> > > update metadata requests to the brokers
> > > - If a timeout error occurs, the topic could still be created
> > > successfully at a later time. Its up to the client to query for
> > the state
> > > at that point.
> > > 6. Setting a timeout <= 0 will validate arguments and trigger the
> > > create topics and return immediately.
> > > - This is essentially the fully asynchronous mode we have in the
> > > Zookeeper tools today.
> > > - The error code in the response will either contain an argument
> > > validation exception or a timeout exception. If you receive a
> > timeout
> > > exception, because you asked for 0 timeout, you can assume the
> > message was
> > > valid and the topic creation was triggered.
> > > 7. The request is not transactional.
> > > 1. If an error occurs on one topic, the others could still be
> > > created.
> > > 2. Errors are reported independently.
> > >
> > > QA:
> > >
> > > - Why is CreateTopicsRequest a batch request?
> > > - Scenarios where tools or admins want to create many topics
> should
> > > be able to with fewer requests
> > > - Example: MirrorMaker may want to create the topics downstream
> > > - What happens if some topics error immediately? Will it
> > > return immediately?
> > > - The request will block until all topics have either been
> created,
> > > errors, or the timeout has been hit
> > > - There is no "short circuiting" where 1 error stops the other
> > > topics from being created
> > > - Why implement "partial blocking" instead of fully async or fully
> > > consistent?
> > > - See Cluster Consistent Blocking
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> > >
> > > below
> > > - Why require the request to go to the controller?
> > > - The controller is responsible for the cluster metadata and its
> > > propagation
> > > - See Request Forwarding
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> > >
> > > below
> > >
> > > Create Topics Response
> > >
> > >
> > >
> > > CreateTopics Response (Version: 0) => [topic_error_codes]
> > > topic_error_codes => topic error_code
> > > topic => STRING
> > > error_code => INT16
> > >
> > > CreateTopicsResponse contains a map between topic and topic creation
> > > result error code (see New Protocol Errors
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> > >
> > > ).
> > >
> >
> > The KIP is available here for reference (linked to the Create Topics
> schema
> > section):
> > *
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> > >*
> >
> > A pull request is available implementing the proposed changes here:
> > https://github.com/apache/kafka/pull/1489
> >
> > Here is a link to the past discussion on the mailing list:
> > *
> >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > <
> >
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> > >*
> >
> > Thank you,
> > Grant
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>
--
-- Guozhang
Re: [VOTE] KIP-4 Create Topics Schema
Posted by Ismael Juma <is...@juma.me.uk>.
+1 (binding)
On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <gh...@cloudera.com> wrote:
> I would like to initiate the voting process for the "KIP-4 Create Topics
> Schema changes". This is not a vote for all of KIP-4, but specifically for
> the create topics changes. I have included the exact changes below for
> clarity:
> >
> > Create Topics Request (KAFKA-2945
> > <https://issues.apache.org/jira/browse/KAFKA-2945>)
> >
> > CreateTopics Request (Version: 0) => [create_topic_requests] timeout
> > create_topic_requests => topic num_partitions replication_factor
> [replica_assignment] [configs]
> > topic => STRING
> > num_partitions => INT32
> > replication_factor => INT16
> > replica_assignment => partition_id [replicas]
> > partition_id => INT32
> > replicas => INT32
> > configs => config_key config_value
> > config_key => STRING
> > config_value => STRING
> > timeout => INT32
> >
> > CreateTopicsRequest is a batch request to initiate topic creation with
> > either predefined or automatic replica assignment and optionally topic
> > configuration.
> >
> > Request semantics:
> >
> > 1. Must be sent to the controller broker
> > 2. If there are multiple instructions for the same topic in one
> > request an InvalidRequestException will be logged on the broker and
> the
> > client will be disconnected.
> > - This is because the list of topics is modeled server side as a
> > map with TopicName as the key
> > 3. The principal must be authorized to the "Create" Operation on the
> > "Cluster" resource to create topics.
> > - Unauthorized requests will receive a
> ClusterAuthorizationException
> > 4.
> >
> > Only one from ReplicaAssignment or (num_partitions +
> replication_factor
> > ), can be defined in one instruction.
> > - If both parameters are specified an InvalidRequestException will be
> > logged on the broker and the client will be disconnected.
> > - In the case ReplicaAssignment is defined number of partitions and
> > replicas will be calculated from the supplied replica_assignment.
> > - In the case of defined (num_partitions + replication_factor)
> > replica assignment will be automatically generated by the server.
> > - One or the other must be defined. The existing broker side auto
> > create defaults will not be used
> > (default.replication.factor, num.partitions). The client
> implementation can
> > have defaults for these options when generating the messages.
> > - The first replica in [replicas] is assumed to be the preferred
> > leader. This matches current behavior elsewhere.
> > 5. Setting a timeout > 0 will allow the request to block until the
> > topic metadata is "complete" on the controller node.
> > - Complete means the local topic metadata cache been completely
> > populated and all partitions have leaders
> > - The topic metadata is updated when the controller sends out
> > update metadata requests to the brokers
> > - If a timeout error occurs, the topic could still be created
> > successfully at a later time. Its up to the client to query for
> the state
> > at that point.
> > 6. Setting a timeout <= 0 will validate arguments and trigger the
> > create topics and return immediately.
> > - This is essentially the fully asynchronous mode we have in the
> > Zookeeper tools today.
> > - The error code in the response will either contain an argument
> > validation exception or a timeout exception. If you receive a
> timeout
> > exception, because you asked for 0 timeout, you can assume the
> message was
> > valid and the topic creation was triggered.
> > 7. The request is not transactional.
> > 1. If an error occurs on one topic, the others could still be
> > created.
> > 2. Errors are reported independently.
> >
> > QA:
> >
> > - Why is CreateTopicsRequest a batch request?
> > - Scenarios where tools or admins want to create many topics should
> > be able to with fewer requests
> > - Example: MirrorMaker may want to create the topics downstream
> > - What happens if some topics error immediately? Will it
> > return immediately?
> > - The request will block until all topics have either been created,
> > errors, or the timeout has been hit
> > - There is no "short circuiting" where 1 error stops the other
> > topics from being created
> > - Why implement "partial blocking" instead of fully async or fully
> > consistent?
> > - See Cluster Consistent Blocking
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking
> >
> > below
> > - Why require the request to go to the controller?
> > - The controller is responsible for the cluster metadata and its
> > propagation
> > - See Request Forwarding
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request
> >
> > below
> >
> > Create Topics Response
> >
> >
> >
> > CreateTopics Response (Version: 0) => [topic_error_codes]
> > topic_error_codes => topic error_code
> > topic => STRING
> > error_code => INT16
> >
> > CreateTopicsResponse contains a map between topic and topic creation
> > result error code (see New Protocol Errors
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors
> >
> > ).
> >
>
> The KIP is available here for reference (linked to the Create Topics schema
> section):
> *
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)
> >*
>
> A pull request is available implementing the proposed changes here:
> https://github.com/apache/kafka/pull/1489
>
> Here is a link to the past discussion on the mailing list:
> *
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> <
> http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
> >*
>
> Thank you,
> Grant
> --
> Grant Henke
> Software Engineer | Cloudera
> grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>