KIP Call Notes

[Gwen Shapira <gs...@cloudera.com>]

Shortest KIP discussion ever :)

Here are my notes. Feel free to correct or expand:

SSL: Harsh is rebasing after Kafka-1928, and resolving an issue with
support for specific API

KIP-4  - ISR in Metadata request: We prefer to preserve this information.

KIP-25 - System Tests: Geoffrey posted an update with discussion
summary, there were no objections to the proposal. We will wait a day
before starting a vote.

New consumer - Separate the API changes for metadata to a new JIRA.
Since the new consumer is still work in progress, there's no need for
KIP for every modification.

Re: KIP Call Notes

[Jun Rao <ju...@confluent.io>]

Jayesh,

Most of the metadata you mentioned already exist. For example, ISR is
already in the TopicMetadataResponse and the high water mark can be
returned from the OffsetRequest. The details of the current wire protocol
can be found at
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol

We have another KIP (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-24+-+Remove+ISR+information+from+TopicMetadataRequest+and+add+broker+level+metadata+request)
to add a new request for collecting some broker level info. There will be a
discussion thread on this at some point that you can chime in.

Thanks,

Jun


On Wed, Jun 10, 2015 at 8:42 AM, Jayesh Thakrar <j_thakrar@yahoo.com.invalid
> wrote:

> Our company (Conversant Media) became a Kafka user pretty recently (about
> 3 months ago) and I have been following the user and dev group for the past
> month, so could have missed the discussions on KIP-4.
> In looking up the current command line capabilities in Kafka 0.8.2.1 (not
> the latest release) it seems we are missing a couple of pieces.
> Ideally, I am looking at getting the following kind of information
> Describe a "Kafka cluster" (example) -kafka cluster metadata = {
>   "brokers" : [ { "id" : 0, attributes from broker Json schema}, { "id" :
> 1, attributes from broker Json schema}, ...]  "topics" : [ { "name" :
> "topic1", attributes from topic and parition and ISR and leader information
> }, { "name" : "topic2", attributes from topic and parition and ISR and
> leader information } ] }
>
> Topic metrics = atleast some kind of message high water mark from the
> leaders for each topic partition
> Describe consumers (example) -Consumer metadata = {     [ "group" :
> "group_id1", "client" : "client_id1", [ list of subscribed topics ]  ],
> ......}
> Having such information will make it easy to build monitoring tools and
> also aid troubleshooting and sizing/configuration.
> Is discussing these things as part of KIP the right process or is there
> some other forum?
> Thanks,Jayesh Thakrar
>
>
>       From: Gwen Shapira <gs...@cloudera.com>
>  To: "dev@kafka.apache.org" <de...@kafka.apache.org>
>  Sent: Tuesday, June 9, 2015 1:19 PM
>  Subject: KIP Call Notes
>
> Shortest KIP discussion ever :)
>
> Here are my notes. Feel free to correct or expand:
>
> SSL: Harsh is rebasing after Kafka-1928, and resolving an issue with
> support for specific API
>
> KIP-4  - ISR in Metadata request: We prefer to preserve this information.
>
> KIP-25 - System Tests: Geoffrey posted an update with discussion
> summary, there were no objections to the proposal. We will wait a day
> before starting a vote.
>
> New consumer - Separate the API changes for metadata to a new JIRA.
> Since the new consumer is still work in progress, there's no need for
> KIP for every modification.
>
>
>
>

Re: KIP Call Notes

[Jayesh Thakrar <j_...@yahoo.com.INVALID>]

Our company (Conversant Media) became a Kafka user pretty recently (about 3 months ago) and I have been following the user and dev group for the past month, so could have missed the discussions on KIP-4.
In looking up the current command line capabilities in Kafka 0.8.2.1 (not the latest release) it seems we are missing a couple of pieces.
Ideally, I am looking at getting the following kind of information
Describe a "Kafka cluster" (example) -kafka cluster metadata = {
  "brokers" : [ { "id" : 0, attributes from broker Json schema}, { "id" : 1, attributes from broker Json schema}, ...]  "topics" : [ { "name" : "topic1", attributes from topic and parition and ISR and leader information }, { "name" : "topic2", attributes from topic and parition and ISR and leader information } ] }

Topic metrics = atleast some kind of message high water mark from the leaders for each topic partition
Describe consumers (example) -Consumer metadata = {     [ "group" : "group_id1", "client" : "client_id1", [ list of subscribed topics ]  ], ......}
Having such information will make it easy to build monitoring tools and also aid troubleshooting and sizing/configuration.
Is discussing these things as part of KIP the right process or is there some other forum?
Thanks,Jayesh Thakrar


      From: Gwen Shapira <gs...@cloudera.com>
 To: "dev@kafka.apache.org" <de...@kafka.apache.org> 
 Sent: Tuesday, June 9, 2015 1:19 PM
 Subject: KIP Call Notes
   
Shortest KIP discussion ever :)

Here are my notes. Feel free to correct or expand:

SSL: Harsh is rebasing after Kafka-1928, and resolving an issue with
support for specific API

KIP-4  - ISR in Metadata request: We prefer to preserve this information.

KIP-25 - System Tests: Geoffrey posted an update with discussion
summary, there were no objections to the proposal. We will wait a day
before starting a vote.

New consumer - Separate the API changes for metadata to a new JIRA.
Since the new consumer is still work in progress, there's no need for
KIP for every modification.