You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Otis Gospodnetic <ot...@gmail.com> on 2014/09/16 23:09:57 UTC

Re: MBeans, dashes, underscores, and KAFKA-1481

Hi,

So maybe I should I should have asked the Q explicitly:
Could we commit the patch from
https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope, it's
clear what problems the current MBean names can cause?

Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/



On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> *Problem:*
> Some Kafka 0.8.x MBeans have names composed of things like <consumer
> group>-<topic>-<metric name>.  Note how dashes are used as delimiters.
>  When <consumer group> and <topic> don't contain the delimiter character
> all is good if you want to extract parts of this MBean name by simply
> splitting on the delimiter character.  The problem is that dashes are
> allowed in topic and group names, so this splitting doesn't work.
> Moreover, underscores are also used as delimiters, and they can also be
> used in things like topic names.
>
> *Example*:
> This MBean's name is composed of <consumer group>-<topic>-BytesPerSec:
>
> kafka.consumer:type="ConsumerTopicMetrics", name="*myGroup**-myTopic**-*
> BytesPerSec"
>
> Here we can actually split on "-" and extract all 3 parts from the MBean
> name::
> * consumer group ('*myGroup*')
> * topic ('*myTopic*')
> * metric (‘BytesPerSec’)
>
> All good!
>
> But imagine if I named the group: *my-Group*
> And if I named the topic: *my-Topic*
>
> Then we'd have:
> kafka.consumer:type="ConsumerTopicMetrics", name="*my-Group**-my-Topic**-*
> BytesPerSec"
>
> Now splitting on "-" would no longer work!  To extract "my-Group" and
> "my-Topic" and "BytesPerSec" parts I would have to know the specific group
> name and topic name to look for and could not use generic approach of just
> splitting the MBean name on the delimiter.
>
> *Solution*:
> The patch in https://issues.apache.org/jira/browse/KAFKA-1481 replaces
> all _ and - characters where they are used as delimiters in MBean names
> with a "|" character.  Because the "I" character is not allowed in topic
> names, consumer groups, host names, splitting on this new and unified
> delimiter works.
>
> I hope this explains the problem, the solution, and that this can make it
> in the next 0.8.x.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Metrics-core's default jmx name has quotes. To add the tags, we are
specifying the jmx name explicitly. So, it's possible to get rid of the
quotes.

Thanks,

Jun

On Fri, Oct 17, 2014 at 1:52 PM, VERMEERBERGEN Alexandre <
Alexandre.VERMEERBERGEN@3ds.com> wrote:

> +1 as well for getting rid of the quotes. Isn't it an inheritance of
> metrics-core? Anyway, hope Kafka devs will find a way to get rid of these!
>
> Cordialement / Best Regards,
> Alexandre VERMEERBERGEN
> R&D Cloud Service Supervision Development Director
>
> –––––––––––––––––––––––
> Office: +33 1 6162 4992
> alexandre.vermeerbergen@3ds.com
> http://www.3ds.com
> –––––––––––––––––––––––
> Dassault Systèmes | 10 rue Marcel Dassault, CS 40501 | 78946
> Vélizy-Villacoublay Cedex | France
>
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Friday, October 17, 2014 22:49
> To: users@kafka.apache.org
> Subject: Re: MBeans, dashes, underscores, and KAFKA-1481
>
> +1 on getting rid of the quotes.
>
> On Fri, Oct 17, 2014 at 12:31 PM, Magnus Spångdal <
> magnus.spangdal@deltaprojects.com> wrote:
>
> > +1 to get rid of quotes, thanks!
> >
> >
> >
> >
> >
> >
> > —
> > Sent from Mailbox
> >
> > On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Hi, everyone,
> > > We are fixing the mbean names in kafka-1482, by adding separate
> > > explicit tags in the name for things like clientId and topic.
> > > Another thing that some people have complained before is that we use
> quotes in the jmx name.
> > > Should we also just get rid of the quotes as part of kafka-1482? So,
> > > instead of
> > >
> "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > > we will have
> > >
> > > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-
> > > 1
> > > Thanks,
> > > Jun
> > > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede
> > > <ne...@gmail.com>
> > > wrote:
> > >> I am going to vote for 1482 to be included in 0.8.2, if we have a
> > >> patch submitted in a week. I think we've had this JIRA opened for
> > >> too long
> > and we
> > >> held people back so it's only fair to release this.
> > >>
> > >> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
> > >>
> > >> > Otis,
> > >> >
> > >> > Just have the patch ready asap. We can make a call then.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> > >> > otis.gospodnetic@gmail.com
> > >> > > wrote:
> > >> >
> > >> > > Hi Jun,
> > >> > >
> > >> > > Would by the end of next week be acceptable for 0.8.2?
> > >> > >
> > >> > > Thanks,
> > >> > > Otis
> > >> > > --
> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > >> > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > >
> > >> > >
> > >> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> > >> > >
> > >> > > > Otis,
> > >> > > >
> > >> > > > Yes, if you guys can help provide a patch in a few days, we
> > >> > > > can
> > >> > probably
> > >> > > > get it to the 0.8.2 release.
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> > >> > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > >
> > >> > > > > Hi Jun,
> > >> > > > >
> > >> > > > > I think your MBean renaming approach will work.  I see
> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix
> > Version
> > >> > > 0.8.2,
> > >> > > > > but
> > >> > > > > is not marked as a Blocker.  We'd love to get the MBeans
> > >> > > > > fixed
> > so
> > >> > this
> > >> > > > > makes it in 0.8.2 release.  Do you know if this is on
> > >> > > > > anyone's
> > >> plate
> > >> > > (the
> > >> > > > > issue is currently Unassigned)?  If not, should we provide
> > >> > > > > a new
> > >> > patch
> > >> > > > that
> > >> > > > > uses your approach?
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Otis
> > >> > > > > --
> > >> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > >> > Management
> > >> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > >
> > >> > > > >
> > >> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
> > wrote:
> > >> > > > >
> > >> > > > > > Otis,
> > >> > > > > >
> > >> > > > > > In kafka-1481, we will have to change the mbean names (at
> > least
> > >> the
> > >> > > > ones
> > >> > > > > > with clientid and topic) anyway. Using the name/value
> > >> > > > > > pair in
> > the
> > >> > > mbean
> > >> > > > > > name allows us to do this in a cleaner way. Yes, "," is
> > >> > > > > > not
> > >> allowed
> > >> > > in
> > >> > > > > > clientid or topic.
> > >> > > > > >
> > >> > > > > > Bhavesh,
> > >> > > > > >
> > >> > > > > > Yes, I was thinking of making changes in the new metrics
> > package.
> > >> > > > > Something
> > >> > > > > > like allowing the sensor names to have name/value pairs.
> > >> > > > > > The
> > jmx
> > >> > > names
> > >> > > > > will
> > >> > > > > > just follow accordingly. This is probably cleaner than
> > >> > > > > > doing
> > the
> > >> > > > > escaping.
> > >> > > > > > Also, the metric names are more intuitive (otherwise, you
> > have to
> > >> > > know
> > >> > > > > > which part is the clientid and which part is the topic).
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > Jun
> > >> > > > > >
> > >> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > >> > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > >
> > >> > > > > > > Hi Jun,
> > >> > > > > > >
> > >> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao
> > >> > > > > > > <junrao@gmail.com
> > >
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Bhavesh,
> > >> > > > > > > >
> > >> > > > > > > > Yes, allowing dot in clientId and topic makes it a
> > >> > > > > > > > bit
> > harder
> > >> > to
> > >> > > > > define
> > >> > > > > > > the
> > >> > > > > > > > JMX bean names. I see a couple of solutions here.
> > >> > > > > > > >
> > >> > > > > > > > 1. Disable dot in clientId and topic names. The issue
> > >> > > > > > > > is
> > that
> > >> > dot
> > >> > > > may
> > >> > > > > > > > already be used in existing deployment.
> > >> > > > > > > >
> > >> > > > > > > > 2. We can represent the JMX bean name differently in
> > >> > > > > > > > the
> > new
> > >> > > > > producer.
> > >> > > > > > > > Instead of
> > >> > > > > > > >   kafka.producer.myclientid:type=mytopic
> > >> > > > > > > > we could change it to
> > >> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > >> > > > > > > >
> > >> > > > > > > > I felt that option 2 is probably better since it
> > >> > > > > > > > doesn't
> > >> affect
> > >> > > > > > existing
> > >> > > > > > > > users.
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > > If it doesn't affect existing users, great.
> > >> > > > > > >
> > >> > > > > > > If you are saying that each "piece" of MBean name could
> > >> > > > > > > be
> > >> > > expressed
> > >> > > > as
> > >> > > > > > > name=value pair, with something like "," (forbidden in
> > >> > > > > > > host
> > >> > names,
> > >> > > > > topic
> > >> > > > > > > names, client IDs, etc. I assume?) then yes, I think
> > >> > > > > > > this
> > would
> > >> > be
> > >> > > > > easier
> > >> > > > > > > to parse and it would be easier for people to
> > >> > > > > > > understand
> > what
> > >> is
> > >> > > > what.
> > >> > > > > > >
> > >> > > > > > > Otis
> > >> > > > > > > --
> > >> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> > >> > > > > > > Log
> > >> > > > Management
> > >> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Otis,
> > >> > > > > > > >
> > >> > > > > > > > We probably can also use option 2 to address KAFKA-1481.
> > For
> > >> > > > > > > topic/clientid
> > >> > > > > > > > specific metrics, we could explicitly specify the
> > >> > > > > > > > metric
> > name
> > >> > so
> > >> > > > that
> > >> > > > > > it
> > >> > > > > > > > contains "topic=mytopic,clientid=myclientid". That
> > >> > > > > > > > seems
> > to
> > >> be
> > >> > a
> > >> > > > much
> > >> > > > > > > > cleaner way than having all parts included in a
> > >> > > > > > > > single
> > string
> > >> > > > > separated
> > >> > > > > > > by
> > >> > > > > > > > '|'.
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > >
> > >> > > > > > > > Jun
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > >> > > > > > > > mistry.p.bhavesh@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > HI Otis,
> > >> > > > > > > > >
> > >> > > > > > > > > What is migration path ?  If topic with special
> > >> > > > > > > > > chars
> > >> exists
> > >> > > > > already(
> > >> > > > > > > > > ".","-","|" etc)  in previous version of
> > producer/consumer
> > >> of
> > >> > > > > Kafka,
> > >> > > > > > > what
> > >> > > > > > > > > happens after the upgrade new producer or consumer
> > (kafka
> > >> > > > version)
> > >> > > > > ?
> > >> > > > > > > > Also,
> > >> > > > > > > > > in new producer API (Kafka Trunk), does this
> > >> > > > > > > > > enforce the
> > >> rule
> > >> > > > about
> > >> > > > > > > > client
> > >> > > > > > > > > id as well ?
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > >
> > >> > > > > > > > > Bhavesh
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > >> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi,
> > >> > > > > > > > > >
> > >> > > > > > > > > > So maybe I should I should have asked the Q
> > explicitly:
> > >> > > > > > > > > > Could we commit the patch from
> > >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481
> > >> > > > > > > > > > now
> > >> > that, I
> > >> > > > > hope,
> > >> > > > > > > > it's
> > >> > > > > > > > > > clear what problems the current MBean names can
> cause?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Otis
> > >> > > > > > > > > > --
> > >> > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> > Centralized
> > >> Log
> > >> > > > > > > Management
> > >> > > > > > > > > > Solr & Elasticsearch Support *
> > >> > > > > > > > > > http://sematext.com/
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis
> > >> > > > > > > > > > Gospodnetic < otis.gospodnetic@gmail.com> wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Problem:*
> > >> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of
> > things
> > >> > like
> > >> > > > > > > <consumer
> > >> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes
> > >> > > > > > > > > > > group>are
> > used
> > >> > as
> > >> > > > > > > > delimiters.
> > >> > > > > > > > > > >  When <consumer group> and <topic> don't
> > >> > > > > > > > > > > contain the
> > >> > > > delimiter
> > >> > > > > > > > > character
> > >> > > > > > > > > > > all is good if you want to extract parts of
> > >> > > > > > > > > > > this
> > MBean
> > >> > name
> > >> > > > by
> > >> > > > > > > simply
> > >> > > > > > > > > > > splitting on the delimiter character.  The
> > >> > > > > > > > > > > problem
> > is
> > >> > that
> > >> > > > > dashes
> > >> > > > > > > are
> > >> > > > > > > > > > > allowed in topic and group names, so this
> > >> > > > > > > > > > > splitting
> > >> > doesn't
> > >> > > > > work.
> > >> > > > > > > > > > > Moreover, underscores are also used as
> > >> > > > > > > > > > > delimiters,
> > and
> > >> > they
> > >> > > > can
> > >> > > > > > > also
> > >> > > > > > > > be
> > >> > > > > > > > > > > used in things like topic names.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Example*:
> > >> > > > > > > > > > > This MBean's name is composed of <consumer
> > >> > > > > > > > group>-<topic>-BytesPerSec:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > >> > > > > > > > > name="*myGroup**-myTopic**-*
> > >> > > > > > > > > > > BytesPerSec"
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Here we can actually split on "-" and extract
> > >> > > > > > > > > > > all 3
> > >> parts
> > >> > > > from
> > >> > > > > > the
> > >> > > > > > > > > MBean
> > >> > > > > > > > > > > name::
> > >> > > > > > > > > > > * consumer group ('*myGroup*')
> > >> > > > > > > > > > > * topic ('*myTopic*')
> > >> > > > > > > > > > > * metric (‘BytesPerSec’)
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > All good!
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > But imagine if I named the group: *my-Group*
> > >> > > > > > > > > > > And if I named the topic: *my-Topic*
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Then we'd have:
> > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > >> > > > > > > > > > name="*my-Group**-my-Topic**-*
> > >> > > > > > > > > > > BytesPerSec"
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Now splitting on "-" would no longer work!  To
> > extract
> > >> > > > > "my-Group"
> > >> > > > > > > and
> > >> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have
> > >> > > > > > > > > > > to
> > know
> > >> > the
> > >> > > > > > > specific
> > >> > > > > > > > > > group
> > >> > > > > > > > > > > name and topic name to look for and could not
> > >> > > > > > > > > > > use
> > >> generic
> > >> > > > > > approach
> > >> > > > > > > of
> > >> > > > > > > > > > just
> > >> > > > > > > > > > > splitting the MBean name on the delimiter.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Solution*:
> > >> > > > > > > > > > > The patch in
> > >> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> > >> > > > > > > > replaces
> > >> > > > > > > > > > > all _ and - characters where they are used as
> > >> delimiters
> > >> > in
> > >> > > > > MBean
> > >> > > > > > > > names
> > >> > > > > > > > > > > with a "|" character.  Because the "I"
> > >> > > > > > > > > > > character is
> > not
> > >> > > > allowed
> > >> > > > > > in
> > >> > > > > > > > > topic
> > >> > > > > > > > > > > names, consumer groups, host names, splitting
> > >> > > > > > > > > > > on
> > this
> > >> new
> > >> > > and
> > >> > > > > > > unified
> > >> > > > > > > > > > > delimiter works.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I hope this explains the problem, the solution,
> > >> > > > > > > > > > > and
> > >> that
> > >> > > this
> > >> > > > > can
> > >> > > > > > > > make
> > >> > > > > > > > > it
> > >> > > > > > > > > > > in the next 0.8.x.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Otis
> > >> > > > > > > > > > > --
> > >> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> > Centralized
> > >> > Log
> > >> > > > > > > > Management
> > >> > > > > > > > > > > Solr & Elasticsearch Support *
> > >> > > > > > > > > > > http://sematext.com/
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential and/or
> privileged.
>
> If you are not one of the named recipients or have received this email in
> error,
>
> (i) you should not read, disclose, or copy it,
>
> (ii) please notify sender of your receipt by reply email and delete this
> email and all attachments,
>
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
>
> For other languages, go to http://www.3ds.com/terms/email-disclaimer
>

RE: MBeans, dashes, underscores, and KAFKA-1481

Posted by VERMEERBERGEN Alexandre <Al...@3ds.com>.

+1 as well for getting rid of the quotes. Isn't it an inheritance of metrics-core? Anyway, hope Kafka devs will find a way to get rid of these!

Cordialement / Best Regards,
Alexandre VERMEERBERGEN
R&D Cloud Service Supervision Development Director

–––––––––––––––––––––––
Office: +33 1 6162 4992
alexandre.vermeerbergen@3ds.com
http://www.3ds.com
–––––––––––––––––––––––
Dassault Systèmes | 10 rue Marcel Dassault, CS 40501 | 78946 Vélizy-Villacoublay Cedex | France



-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Friday, October 17, 2014 22:49
To: users@kafka.apache.org
Subject: Re: MBeans, dashes, underscores, and KAFKA-1481

+1 on getting rid of the quotes.

On Fri, Oct 17, 2014 at 12:31 PM, Magnus Spångdal < magnus.spangdal@deltaprojects.com> wrote:

> +1 to get rid of quotes, thanks!
>
>
>
>
>
>
> —
> Sent from Mailbox
>
> On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Hi, everyone,
> > We are fixing the mbean names in kafka-1482, by adding separate
> > explicit tags in the name for things like clientId and topic.
> > Another thing that some people have complained before is that we use quotes in the jmx name.
> > Should we also just get rid of the quotes as part of kafka-1482? So,
> > instead of
> >    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > we will have
> >
> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-
> > 1
> > Thanks,
> > Jun
> > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede
> > <ne...@gmail.com>
> > wrote:
> >> I am going to vote for 1482 to be included in 0.8.2, if we have a
> >> patch submitted in a week. I think we've had this JIRA opened for
> >> too long
> and we
> >> held people back so it's only fair to release this.
> >>
> >> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
> >>
> >> > Otis,
> >> >
> >> > Just have the patch ready asap. We can make a call then.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> >> > otis.gospodnetic@gmail.com
> >> > > wrote:
> >> >
> >> > > Hi Jun,
> >> > >
> >> > > Would by the end of next week be acceptable for 0.8.2?
> >> > >
> >> > > Thanks,
> >> > > Otis
> >> > > --
> >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> >> > > Solr & Elasticsearch Support * http://sematext.com/
> >> > >
> >> > >
> >> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> >> > >
> >> > > > Otis,
> >> > > >
> >> > > > Yes, if you guys can help provide a patch in a few days, we
> >> > > > can
> >> > probably
> >> > > > get it to the 0.8.2 release.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> >> > > > otis.gospodnetic@gmail.com> wrote:
> >> > > >
> >> > > > > Hi Jun,
> >> > > > >
> >> > > > > I think your MBean renaming approach will work.  I see
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix
> Version
> >> > > 0.8.2,
> >> > > > > but
> >> > > > > is not marked as a Blocker.  We'd love to get the MBeans
> >> > > > > fixed
> so
> >> > this
> >> > > > > makes it in 0.8.2 release.  Do you know if this is on
> >> > > > > anyone's
> >> plate
> >> > > (the
> >> > > > > issue is currently Unassigned)?  If not, should we provide
> >> > > > > a new
> >> > patch
> >> > > > that
> >> > > > > uses your approach?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Otis
> >> > > > > --
> >> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> > Management
> >> > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
> wrote:
> >> > > > >
> >> > > > > > Otis,
> >> > > > > >
> >> > > > > > In kafka-1481, we will have to change the mbean names (at
> least
> >> the
> >> > > > ones
> >> > > > > > with clientid and topic) anyway. Using the name/value
> >> > > > > > pair in
> the
> >> > > mbean
> >> > > > > > name allows us to do this in a cleaner way. Yes, "," is
> >> > > > > > not
> >> allowed
> >> > > in
> >> > > > > > clientid or topic.
> >> > > > > >
> >> > > > > > Bhavesh,
> >> > > > > >
> >> > > > > > Yes, I was thinking of making changes in the new metrics
> package.
> >> > > > > Something
> >> > > > > > like allowing the sensor names to have name/value pairs.
> >> > > > > > The
> jmx
> >> > > names
> >> > > > > will
> >> > > > > > just follow accordingly. This is probably cleaner than
> >> > > > > > doing
> the
> >> > > > > escaping.
> >> > > > > > Also, the metric names are more intuitive (otherwise, you
> have to
> >> > > know
> >> > > > > > which part is the clientid and which part is the topic).
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > Jun
> >> > > > > >
> >> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> >> > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > >
> >> > > > > > > Hi Jun,
> >> > > > > > >
> >> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao
> >> > > > > > > <junrao@gmail.com
> >
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > Bhavesh,
> >> > > > > > > >
> >> > > > > > > > Yes, allowing dot in clientId and topic makes it a
> >> > > > > > > > bit
> harder
> >> > to
> >> > > > > define
> >> > > > > > > the
> >> > > > > > > > JMX bean names. I see a couple of solutions here.
> >> > > > > > > >
> >> > > > > > > > 1. Disable dot in clientId and topic names. The issue
> >> > > > > > > > is
> that
> >> > dot
> >> > > > may
> >> > > > > > > > already be used in existing deployment.
> >> > > > > > > >
> >> > > > > > > > 2. We can represent the JMX bean name differently in
> >> > > > > > > > the
> new
> >> > > > > producer.
> >> > > > > > > > Instead of
> >> > > > > > > >   kafka.producer.myclientid:type=mytopic
> >> > > > > > > > we could change it to
> >> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> >> > > > > > > >
> >> > > > > > > > I felt that option 2 is probably better since it
> >> > > > > > > > doesn't
> >> affect
> >> > > > > > existing
> >> > > > > > > > users.
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > If it doesn't affect existing users, great.
> >> > > > > > >
> >> > > > > > > If you are saying that each "piece" of MBean name could
> >> > > > > > > be
> >> > > expressed
> >> > > > as
> >> > > > > > > name=value pair, with something like "," (forbidden in
> >> > > > > > > host
> >> > names,
> >> > > > > topic
> >> > > > > > > names, client IDs, etc. I assume?) then yes, I think
> >> > > > > > > this
> would
> >> > be
> >> > > > > easier
> >> > > > > > > to parse and it would be easier for people to
> >> > > > > > > understand
> what
> >> is
> >> > > > what.
> >> > > > > > >
> >> > > > > > > Otis
> >> > > > > > > --
> >> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> >> > > > > > > Log
> >> > > > Management
> >> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > Otis,
> >> > > > > > > >
> >> > > > > > > > We probably can also use option 2 to address KAFKA-1481.
> For
> >> > > > > > > topic/clientid
> >> > > > > > > > specific metrics, we could explicitly specify the
> >> > > > > > > > metric
> name
> >> > so
> >> > > > that
> >> > > > > > it
> >> > > > > > > > contains "topic=mytopic,clientid=myclientid". That
> >> > > > > > > > seems
> to
> >> be
> >> > a
> >> > > > much
> >> > > > > > > > cleaner way than having all parts included in a
> >> > > > > > > > single
> string
> >> > > > > separated
> >> > > > > > > by
> >> > > > > > > > '|'.
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > >
> >> > > > > > > > Jun
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> >> > > > > > > > mistry.p.bhavesh@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > HI Otis,
> >> > > > > > > > >
> >> > > > > > > > > What is migration path ?  If topic with special
> >> > > > > > > > > chars
> >> exists
> >> > > > > already(
> >> > > > > > > > > ".","-","|" etc)  in previous version of
> producer/consumer
> >> of
> >> > > > > Kafka,
> >> > > > > > > what
> >> > > > > > > > > happens after the upgrade new producer or consumer
> (kafka
> >> > > > version)
> >> > > > > ?
> >> > > > > > > > Also,
> >> > > > > > > > > in new producer API (Kafka Trunk), does this
> >> > > > > > > > > enforce the
> >> rule
> >> > > > about
> >> > > > > > > > client
> >> > > > > > > > > id as well ?
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Bhavesh
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> >> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > > So maybe I should I should have asked the Q
> explicitly:
> >> > > > > > > > > > Could we commit the patch from
> >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481
> >> > > > > > > > > > now
> >> > that, I
> >> > > > > hope,
> >> > > > > > > > it's
> >> > > > > > > > > > clear what problems the current MBean names can cause?
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Otis
> >> > > > > > > > > > --
> >> > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> Centralized
> >> Log
> >> > > > > > > Management
> >> > > > > > > > > > Solr & Elasticsearch Support *
> >> > > > > > > > > > http://sematext.com/
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis
> >> > > > > > > > > > Gospodnetic < otis.gospodnetic@gmail.com> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi,
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Problem:*
> >> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of
> things
> >> > like
> >> > > > > > > <consumer
> >> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes
> >> > > > > > > > > > > group>are
> used
> >> > as
> >> > > > > > > > delimiters.
> >> > > > > > > > > > >  When <consumer group> and <topic> don't
> >> > > > > > > > > > > contain the
> >> > > > delimiter
> >> > > > > > > > > character
> >> > > > > > > > > > > all is good if you want to extract parts of
> >> > > > > > > > > > > this
> MBean
> >> > name
> >> > > > by
> >> > > > > > > simply
> >> > > > > > > > > > > splitting on the delimiter character.  The
> >> > > > > > > > > > > problem
> is
> >> > that
> >> > > > > dashes
> >> > > > > > > are
> >> > > > > > > > > > > allowed in topic and group names, so this
> >> > > > > > > > > > > splitting
> >> > doesn't
> >> > > > > work.
> >> > > > > > > > > > > Moreover, underscores are also used as
> >> > > > > > > > > > > delimiters,
> and
> >> > they
> >> > > > can
> >> > > > > > > also
> >> > > > > > > > be
> >> > > > > > > > > > > used in things like topic names.
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Example*:
> >> > > > > > > > > > > This MBean's name is composed of <consumer
> >> > > > > > > > group>-<topic>-BytesPerSec:
> >> > > > > > > > > > >
> >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> >> > > > > > > > > name="*myGroup**-myTopic**-*
> >> > > > > > > > > > > BytesPerSec"
> >> > > > > > > > > > >
> >> > > > > > > > > > > Here we can actually split on "-" and extract
> >> > > > > > > > > > > all 3
> >> parts
> >> > > > from
> >> > > > > > the
> >> > > > > > > > > MBean
> >> > > > > > > > > > > name::
> >> > > > > > > > > > > * consumer group ('*myGroup*')
> >> > > > > > > > > > > * topic ('*myTopic*')
> >> > > > > > > > > > > * metric (‘BytesPerSec’)
> >> > > > > > > > > > >
> >> > > > > > > > > > > All good!
> >> > > > > > > > > > >
> >> > > > > > > > > > > But imagine if I named the group: *my-Group*
> >> > > > > > > > > > > And if I named the topic: *my-Topic*
> >> > > > > > > > > > >
> >> > > > > > > > > > > Then we'd have:
> >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> >> > > > > > > > > > name="*my-Group**-my-Topic**-*
> >> > > > > > > > > > > BytesPerSec"
> >> > > > > > > > > > >
> >> > > > > > > > > > > Now splitting on "-" would no longer work!  To
> extract
> >> > > > > "my-Group"
> >> > > > > > > and
> >> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have
> >> > > > > > > > > > > to
> know
> >> > the
> >> > > > > > > specific
> >> > > > > > > > > > group
> >> > > > > > > > > > > name and topic name to look for and could not
> >> > > > > > > > > > > use
> >> generic
> >> > > > > > approach
> >> > > > > > > of
> >> > > > > > > > > > just
> >> > > > > > > > > > > splitting the MBean name on the delimiter.
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Solution*:
> >> > > > > > > > > > > The patch in
> >> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> >> > > > > > > > replaces
> >> > > > > > > > > > > all _ and - characters where they are used as
> >> delimiters
> >> > in
> >> > > > > MBean
> >> > > > > > > > names
> >> > > > > > > > > > > with a "|" character.  Because the "I"
> >> > > > > > > > > > > character is
> not
> >> > > > allowed
> >> > > > > > in
> >> > > > > > > > > topic
> >> > > > > > > > > > > names, consumer groups, host names, splitting
> >> > > > > > > > > > > on
> this
> >> new
> >> > > and
> >> > > > > > > unified
> >> > > > > > > > > > > delimiter works.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I hope this explains the problem, the solution,
> >> > > > > > > > > > > and
> >> that
> >> > > this
> >> > > > > can
> >> > > > > > > > make
> >> > > > > > > > > it
> >> > > > > > > > > > > in the next 0.8.x.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Otis
> >> > > > > > > > > > > --
> >> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> Centralized
> >> > Log
> >> > > > > > > > Management
> >> > > > > > > > > > > Solr & Elasticsearch Support *
> >> > > > > > > > > > > http://sematext.com/
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.

If you are not one of the named recipients or have received this email in error,

(i) you should not read, disclose, or copy it,

(ii) please notify sender of your receipt by reply email and delete this email and all attachments,

(iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.

For other languages, go to http://www.3ds.com/terms/email-disclaimer

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Daniel Compton <de...@danielcompton.net>.

I'm pretty sure that the quotes are a side effect of using Metrics 2.x. When part of an Mbean name has certain characters, then that part will be wrapped in quotes. This is fixed in Metrics 3. 

--
Daniel

> On 18/10/2014, at 10:03 am, Rajasekar Elango <re...@salesforce.com> wrote:
> 
> +1 on getting rid of quotes in jmx mbeans.
> 
> Thanks,
> Raja.
> 
> On Fri, Oct 17, 2014 at 4:48 PM, Neha Narkhede <ne...@gmail.com>
> wrote:
> 
>> +1 on getting rid of the quotes.
>> 
>> On Fri, Oct 17, 2014 at 12:31 PM, Magnus Spångdal <
>> magnus.spangdal@deltaprojects.com> wrote:
>> 
>>> +1 to get rid of quotes, thanks!
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> —
>>> Sent from Mailbox
>>> 
>>>> On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao <ju...@gmail.com> wrote:
>>>> 
>>>> Hi, everyone,
>>>> We are fixing the mbean names in kafka-1482, by adding separate
>> explicit
>>>> tags in the name for things like clientId and topic. Another thing that
>>>> some people have complained before is that we use quotes in the jmx
>> name.
>>>> Should we also just get rid of the quotes as part of kafka-1482? So,
>>>> instead of
>> "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
>>>> we will have
>> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
>>>> Thanks,
>>>> Jun
>>>> On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <
>> neha.narkhede@gmail.com>
>>>> wrote:
>>>>> I am going to vote for 1482 to be included in 0.8.2, if we have a
>> patch
>>>>> submitted in a week. I think we've had this JIRA opened for too long
>>> and we
>>>>> held people back so it's only fair to release this.
>>>>> 
>>>>>> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
>>>>>> 
>>>>>> Otis,
>>>>>> 
>>>>>> Just have the patch ready asap. We can make a call then.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
>>>>>> otis.gospodnetic@gmail.com
>>>>>>> wrote:
>>>>>> 
>>>>>>> Hi Jun,
>>>>>>> 
>>>>>>> Would by the end of next week be acceptable for 0.8.2?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Otis
>>>>>>> --
>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log
>>> Management
>>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>>> 
>>>>>>> 
>>>>>>>> On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Otis,
>>>>>>>> 
>>>>>>>> Yes, if you guys can help provide a patch in a few days, we can
>>>>>> probably
>>>>>>>> get it to the 0.8.2 release.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
>>>>>>>> otis.gospodnetic@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jun,
>>>>>>>>> 
>>>>>>>>> I think your MBean renaming approach will work.  I see
>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-1481 has Fix
>>> Version
>>>>>>> 0.8.2,
>>>>>>>>> but
>>>>>>>>> is not marked as a Blocker.  We'd love to get the MBeans fixed
>>> so
>>>>>> this
>>>>>>>>> makes it in 0.8.2 release.  Do you know if this is on anyone's
>>>>> plate
>>>>>>> (the
>>>>>>>>> issue is currently Unassigned)?  If not, should we provide a
>> new
>>>>>> patch
>>>>>>>> that
>>>>>>>>> uses your approach?
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Otis
>>>>>>>>> --
>>>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log
>>>>>> Management
>>>>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Otis,
>>>>>>>>>> 
>>>>>>>>>> In kafka-1481, we will have to change the mbean names (at
>>> least
>>>>> the
>>>>>>>> ones
>>>>>>>>>> with clientid and topic) anyway. Using the name/value pair
>> in
>>> the
>>>>>>> mbean
>>>>>>>>>> name allows us to do this in a cleaner way. Yes, "," is not
>>>>> allowed
>>>>>>> in
>>>>>>>>>> clientid or topic.
>>>>>>>>>> 
>>>>>>>>>> Bhavesh,
>>>>>>>>>> 
>>>>>>>>>> Yes, I was thinking of making changes in the new metrics
>>> package.
>>>>>>>>> Something
>>>>>>>>>> like allowing the sensor names to have name/value pairs. The
>>> jmx
>>>>>>> names
>>>>>>>>> will
>>>>>>>>>> just follow accordingly. This is probably cleaner than doing
>>> the
>>>>>>>>> escaping.
>>>>>>>>>> Also, the metric names are more intuitive (otherwise, you
>>> have to
>>>>>>> know
>>>>>>>>>> which part is the clientid and which part is the topic).
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
>>>>>>>>>> otis.gospodnetic@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Jun,
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <
>> junrao@gmail.com
>>>> 
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Bhavesh,
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes, allowing dot in clientId and topic makes it a bit
>>> harder
>>>>>> to
>>>>>>>>> define
>>>>>>>>>>> the
>>>>>>>>>>>> JMX bean names. I see a couple of solutions here.
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. Disable dot in clientId and topic names. The issue is
>>> that
>>>>>> dot
>>>>>>>> may
>>>>>>>>>>>> already be used in existing deployment.
>>>>>>>>>>>> 
>>>>>>>>>>>> 2. We can represent the JMX bean name differently in the
>>> new
>>>>>>>>> producer.
>>>>>>>>>>>> Instead of
>>>>>>>>>>>>  kafka.producer.myclientid:type=mytopic
>>>>>>>>>>>> we could change it to
>>>>>>>>>>>>  kafka.producer:clientId=myclientid,topic=mytopic
>>>>>>>>>>>> 
>>>>>>>>>>>> I felt that option 2 is probably better since it doesn't
>>>>> affect
>>>>>>>>>> existing
>>>>>>>>>>>> users.
>>>>>>>>>>> 
>>>>>>>>>>> If it doesn't affect existing users, great.
>>>>>>>>>>> 
>>>>>>>>>>> If you are saying that each "piece" of MBean name could be
>>>>>>> expressed
>>>>>>>> as
>>>>>>>>>>> name=value pair, with something like "," (forbidden in
>> host
>>>>>> names,
>>>>>>>>> topic
>>>>>>>>>>> names, client IDs, etc. I assume?) then yes, I think this
>>> would
>>>>>> be
>>>>>>>>> easier
>>>>>>>>>>> to parse and it would be easier for people to understand
>>> what
>>>>> is
>>>>>>>> what.
>>>>>>>>>>> 
>>>>>>>>>>> Otis
>>>>>>>>>>> --
>>>>>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized
>> Log
>>>>>>>> Management
>>>>>>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Otis,
>>>>>>>>>>>> 
>>>>>>>>>>>> We probably can also use option 2 to address KAFKA-1481.
>>> For
>>>>>>>>>>> topic/clientid
>>>>>>>>>>>> specific metrics, we could explicitly specify the metric
>>> name
>>>>>> so
>>>>>>>> that
>>>>>>>>>> it
>>>>>>>>>>>> contains "topic=mytopic,clientid=myclientid". That seems
>>> to
>>>>> be
>>>>>> a
>>>>>>>> much
>>>>>>>>>>>> cleaner way than having all parts included in a single
>>> string
>>>>>>>>> separated
>>>>>>>>>>> by
>>>>>>>>>>>> '|'.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
>>>>>>>>>>>> mistry.p.bhavesh@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> HI Otis,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> What is migration path ?  If topic with special chars
>>>>> exists
>>>>>>>>> already(
>>>>>>>>>>>>> ".","-","|" etc)  in previous version of
>>> producer/consumer
>>>>> of
>>>>>>>>> Kafka,
>>>>>>>>>>> what
>>>>>>>>>>>>> happens after the upgrade new producer or consumer
>>> (kafka
>>>>>>>> version)
>>>>>>>>> ?
>>>>>>>>>>>> Also,
>>>>>>>>>>>>> in new producer API (Kafka Trunk), does this enforce
>> the
>>>>> rule
>>>>>>>> about
>>>>>>>>>>>> client
>>>>>>>>>>>>> id as well ?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Bhavesh
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
>>>>>>>>>>>>> otis.gospodnetic@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So maybe I should I should have asked the Q
>>> explicitly:
>>>>>>>>>>>>>> Could we commit the patch from
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-1481
>> now
>>>>>> that, I
>>>>>>>>> hope,
>>>>>>>>>>>> it's
>>>>>>>>>>>>>> clear what problems the current MBean names can
>> cause?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Otis
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Monitoring * Alerting * Anomaly Detection *
>>> Centralized
>>>>> Log
>>>>>>>>>>> Management
>>>>>>>>>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
>>>>>>>>>>>>>> otis.gospodnetic@gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *Problem:*
>>>>>>>>>>>>>>> Some Kafka 0.8.x MBeans have names composed of
>>> things
>>>>>> like
>>>>>>>>>>> <consumer
>>>>>>>>>>>>>>> group>-<topic>-<metric name>.  Note how dashes are
>>> used
>>>>>> as
>>>>>>>>>>>> delimiters.
>>>>>>>>>>>>>>> When <consumer group> and <topic> don't contain
>> the
>>>>>>>> delimiter
>>>>>>>>>>>>> character
>>>>>>>>>>>>>>> all is good if you want to extract parts of this
>>> MBean
>>>>>> name
>>>>>>>> by
>>>>>>>>>>> simply
>>>>>>>>>>>>>>> splitting on the delimiter character.  The problem
>>> is
>>>>>> that
>>>>>>>>> dashes
>>>>>>>>>>> are
>>>>>>>>>>>>>>> allowed in topic and group names, so this
>> splitting
>>>>>> doesn't
>>>>>>>>> work.
>>>>>>>>>>>>>>> Moreover, underscores are also used as delimiters,
>>> and
>>>>>> they
>>>>>>>> can
>>>>>>>>>>> also
>>>>>>>>>>>> be
>>>>>>>>>>>>>>> used in things like topic names.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *Example*:
>>>>>>>>>>>>>>> This MBean's name is composed of <consumer
>>>>>>>>>>>> group>-<topic>-BytesPerSec:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> kafka.consumer:type="ConsumerTopicMetrics",
>>>>>>>>>>>>> name="*myGroup**-myTopic**-*
>>>>>>>>>>>>>>> BytesPerSec"
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Here we can actually split on "-" and extract all
>> 3
>>>>> parts
>>>>>>>> from
>>>>>>>>>> the
>>>>>>>>>>>>> MBean
>>>>>>>>>>>>>>> name::
>>>>>>>>>>>>>>> * consumer group ('*myGroup*')
>>>>>>>>>>>>>>> * topic ('*myTopic*')
>>>>>>>>>>>>>>> * metric (‘BytesPerSec’)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> All good!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> But imagine if I named the group: *my-Group*
>>>>>>>>>>>>>>> And if I named the topic: *my-Topic*
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Then we'd have:
>>>>>>>>>>>>>>> kafka.consumer:type="ConsumerTopicMetrics",
>>>>>>>>>>>>>> name="*my-Group**-my-Topic**-*
>>>>>>>>>>>>>>> BytesPerSec"
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Now splitting on "-" would no longer work!  To
>>> extract
>>>>>>>>> "my-Group"
>>>>>>>>>>> and
>>>>>>>>>>>>>>> "my-Topic" and "BytesPerSec" parts I would have to
>>> know
>>>>>> the
>>>>>>>>>>> specific
>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>> name and topic name to look for and could not use
>>>>> generic
>>>>>>>>>> approach
>>>>>>>>>>> of
>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>> splitting the MBean name on the delimiter.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *Solution*:
>>>>>>>>>>>>>>> The patch in
>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-1481
>>>>>>>>>>>> replaces
>>>>>>>>>>>>>>> all _ and - characters where they are used as
>>>>> delimiters
>>>>>> in
>>>>>>>>> MBean
>>>>>>>>>>>> names
>>>>>>>>>>>>>>> with a "|" character.  Because the "I" character
>> is
>>> not
>>>>>>>> allowed
>>>>>>>>>> in
>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>> names, consumer groups, host names, splitting on
>>> this
>>>>> new
>>>>>>> and
>>>>>>>>>>> unified
>>>>>>>>>>>>>>> delimiter works.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I hope this explains the problem, the solution,
>> and
>>>>> that
>>>>>>> this
>>>>>>>>> can
>>>>>>>>>>>> make
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> in the next 0.8.x.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Otis
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Monitoring * Alerting * Anomaly Detection *
>>> Centralized
>>>>>> Log
>>>>>>>>>>>> Management
>>>>>>>>>>>>>>> Solr & Elasticsearch Support *
>> http://sematext.com/
> 
> 
> 
> -- 
> Thanks,
> Raja.

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Rajasekar Elango <re...@salesforce.com>.

+1 on getting rid of quotes in jmx mbeans.

Thanks,
Raja.

On Fri, Oct 17, 2014 at 4:48 PM, Neha Narkhede <ne...@gmail.com>
wrote:

> +1 on getting rid of the quotes.
>
> On Fri, Oct 17, 2014 at 12:31 PM, Magnus Spångdal <
> magnus.spangdal@deltaprojects.com> wrote:
>
> > +1 to get rid of quotes, thanks!
> >
> >
> >
> >
> >
> >
> > —
> > Sent from Mailbox
> >
> > On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Hi, everyone,
> > > We are fixing the mbean names in kafka-1482, by adding separate
> explicit
> > > tags in the name for things like clientId and topic. Another thing that
> > > some people have complained before is that we use quotes in the jmx
> name.
> > > Should we also just get rid of the quotes as part of kafka-1482? So,
> > > instead of
> > >
> "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > > we will have
> > >
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> > > Thanks,
> > > Jun
> > > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <
> neha.narkhede@gmail.com>
> > > wrote:
> > >> I am going to vote for 1482 to be included in 0.8.2, if we have a
> patch
> > >> submitted in a week. I think we've had this JIRA opened for too long
> > and we
> > >> held people back so it's only fair to release this.
> > >>
> > >> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
> > >>
> > >> > Otis,
> > >> >
> > >> > Just have the patch ready asap. We can make a call then.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> > >> > otis.gospodnetic@gmail.com
> > >> > > wrote:
> > >> >
> > >> > > Hi Jun,
> > >> > >
> > >> > > Would by the end of next week be acceptable for 0.8.2?
> > >> > >
> > >> > > Thanks,
> > >> > > Otis
> > >> > > --
> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > >> > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > >
> > >> > >
> > >> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> > >> > >
> > >> > > > Otis,
> > >> > > >
> > >> > > > Yes, if you guys can help provide a patch in a few days, we can
> > >> > probably
> > >> > > > get it to the 0.8.2 release.
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> > >> > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > >
> > >> > > > > Hi Jun,
> > >> > > > >
> > >> > > > > I think your MBean renaming approach will work.  I see
> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix
> > Version
> > >> > > 0.8.2,
> > >> > > > > but
> > >> > > > > is not marked as a Blocker.  We'd love to get the MBeans fixed
> > so
> > >> > this
> > >> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
> > >> plate
> > >> > > (the
> > >> > > > > issue is currently Unassigned)?  If not, should we provide a
> new
> > >> > patch
> > >> > > > that
> > >> > > > > uses your approach?
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Otis
> > >> > > > > --
> > >> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > >> > Management
> > >> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > >
> > >> > > > >
> > >> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
> > wrote:
> > >> > > > >
> > >> > > > > > Otis,
> > >> > > > > >
> > >> > > > > > In kafka-1481, we will have to change the mbean names (at
> > least
> > >> the
> > >> > > > ones
> > >> > > > > > with clientid and topic) anyway. Using the name/value pair
> in
> > the
> > >> > > mbean
> > >> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
> > >> allowed
> > >> > > in
> > >> > > > > > clientid or topic.
> > >> > > > > >
> > >> > > > > > Bhavesh,
> > >> > > > > >
> > >> > > > > > Yes, I was thinking of making changes in the new metrics
> > package.
> > >> > > > > Something
> > >> > > > > > like allowing the sensor names to have name/value pairs. The
> > jmx
> > >> > > names
> > >> > > > > will
> > >> > > > > > just follow accordingly. This is probably cleaner than doing
> > the
> > >> > > > > escaping.
> > >> > > > > > Also, the metric names are more intuitive (otherwise, you
> > have to
> > >> > > know
> > >> > > > > > which part is the clientid and which part is the topic).
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > Jun
> > >> > > > > >
> > >> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > >> > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > >
> > >> > > > > > > Hi Jun,
> > >> > > > > > >
> > >> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <
> junrao@gmail.com
> > >
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Bhavesh,
> > >> > > > > > > >
> > >> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit
> > harder
> > >> > to
> > >> > > > > define
> > >> > > > > > > the
> > >> > > > > > > > JMX bean names. I see a couple of solutions here.
> > >> > > > > > > >
> > >> > > > > > > > 1. Disable dot in clientId and topic names. The issue is
> > that
> > >> > dot
> > >> > > > may
> > >> > > > > > > > already be used in existing deployment.
> > >> > > > > > > >
> > >> > > > > > > > 2. We can represent the JMX bean name differently in the
> > new
> > >> > > > > producer.
> > >> > > > > > > > Instead of
> > >> > > > > > > >   kafka.producer.myclientid:type=mytopic
> > >> > > > > > > > we could change it to
> > >> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > >> > > > > > > >
> > >> > > > > > > > I felt that option 2 is probably better since it doesn't
> > >> affect
> > >> > > > > > existing
> > >> > > > > > > > users.
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > > If it doesn't affect existing users, great.
> > >> > > > > > >
> > >> > > > > > > If you are saying that each "piece" of MBean name could be
> > >> > > expressed
> > >> > > > as
> > >> > > > > > > name=value pair, with something like "," (forbidden in
> host
> > >> > names,
> > >> > > > > topic
> > >> > > > > > > names, client IDs, etc. I assume?) then yes, I think this
> > would
> > >> > be
> > >> > > > > easier
> > >> > > > > > > to parse and it would be easier for people to understand
> > what
> > >> is
> > >> > > > what.
> > >> > > > > > >
> > >> > > > > > > Otis
> > >> > > > > > > --
> > >> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> Log
> > >> > > > Management
> > >> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Otis,
> > >> > > > > > > >
> > >> > > > > > > > We probably can also use option 2 to address KAFKA-1481.
> > For
> > >> > > > > > > topic/clientid
> > >> > > > > > > > specific metrics, we could explicitly specify the metric
> > name
> > >> > so
> > >> > > > that
> > >> > > > > > it
> > >> > > > > > > > contains "topic=mytopic,clientid=myclientid". That seems
> > to
> > >> be
> > >> > a
> > >> > > > much
> > >> > > > > > > > cleaner way than having all parts included in a single
> > string
> > >> > > > > separated
> > >> > > > > > > by
> > >> > > > > > > > '|'.
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > >
> > >> > > > > > > > Jun
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > >> > > > > > > > mistry.p.bhavesh@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > HI Otis,
> > >> > > > > > > > >
> > >> > > > > > > > > What is migration path ?  If topic with special chars
> > >> exists
> > >> > > > > already(
> > >> > > > > > > > > ".","-","|" etc)  in previous version of
> > producer/consumer
> > >> of
> > >> > > > > Kafka,
> > >> > > > > > > what
> > >> > > > > > > > > happens after the upgrade new producer or consumer
> > (kafka
> > >> > > > version)
> > >> > > > > ?
> > >> > > > > > > > Also,
> > >> > > > > > > > > in new producer API (Kafka Trunk), does this enforce
> the
> > >> rule
> > >> > > > about
> > >> > > > > > > > client
> > >> > > > > > > > > id as well ?
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > >
> > >> > > > > > > > > Bhavesh
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > >> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi,
> > >> > > > > > > > > >
> > >> > > > > > > > > > So maybe I should I should have asked the Q
> > explicitly:
> > >> > > > > > > > > > Could we commit the patch from
> > >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481
> now
> > >> > that, I
> > >> > > > > hope,
> > >> > > > > > > > it's
> > >> > > > > > > > > > clear what problems the current MBean names can
> cause?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Otis
> > >> > > > > > > > > > --
> > >> > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> > Centralized
> > >> Log
> > >> > > > > > > Management
> > >> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > >> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Problem:*
> > >> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of
> > things
> > >> > like
> > >> > > > > > > <consumer
> > >> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are
> > used
> > >> > as
> > >> > > > > > > > delimiters.
> > >> > > > > > > > > > >  When <consumer group> and <topic> don't contain
> the
> > >> > > > delimiter
> > >> > > > > > > > > character
> > >> > > > > > > > > > > all is good if you want to extract parts of this
> > MBean
> > >> > name
> > >> > > > by
> > >> > > > > > > simply
> > >> > > > > > > > > > > splitting on the delimiter character.  The problem
> > is
> > >> > that
> > >> > > > > dashes
> > >> > > > > > > are
> > >> > > > > > > > > > > allowed in topic and group names, so this
> splitting
> > >> > doesn't
> > >> > > > > work.
> > >> > > > > > > > > > > Moreover, underscores are also used as delimiters,
> > and
> > >> > they
> > >> > > > can
> > >> > > > > > > also
> > >> > > > > > > > be
> > >> > > > > > > > > > > used in things like topic names.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Example*:
> > >> > > > > > > > > > > This MBean's name is composed of <consumer
> > >> > > > > > > > group>-<topic>-BytesPerSec:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > >> > > > > > > > > name="*myGroup**-myTopic**-*
> > >> > > > > > > > > > > BytesPerSec"
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Here we can actually split on "-" and extract all
> 3
> > >> parts
> > >> > > > from
> > >> > > > > > the
> > >> > > > > > > > > MBean
> > >> > > > > > > > > > > name::
> > >> > > > > > > > > > > * consumer group ('*myGroup*')
> > >> > > > > > > > > > > * topic ('*myTopic*')
> > >> > > > > > > > > > > * metric (‘BytesPerSec’)
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > All good!
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > But imagine if I named the group: *my-Group*
> > >> > > > > > > > > > > And if I named the topic: *my-Topic*
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Then we'd have:
> > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > >> > > > > > > > > > name="*my-Group**-my-Topic**-*
> > >> > > > > > > > > > > BytesPerSec"
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Now splitting on "-" would no longer work!  To
> > extract
> > >> > > > > "my-Group"
> > >> > > > > > > and
> > >> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to
> > know
> > >> > the
> > >> > > > > > > specific
> > >> > > > > > > > > > group
> > >> > > > > > > > > > > name and topic name to look for and could not use
> > >> generic
> > >> > > > > > approach
> > >> > > > > > > of
> > >> > > > > > > > > > just
> > >> > > > > > > > > > > splitting the MBean name on the delimiter.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Solution*:
> > >> > > > > > > > > > > The patch in
> > >> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> > >> > > > > > > > replaces
> > >> > > > > > > > > > > all _ and - characters where they are used as
> > >> delimiters
> > >> > in
> > >> > > > > MBean
> > >> > > > > > > > names
> > >> > > > > > > > > > > with a "|" character.  Because the "I" character
> is
> > not
> > >> > > > allowed
> > >> > > > > > in
> > >> > > > > > > > > topic
> > >> > > > > > > > > > > names, consumer groups, host names, splitting on
> > this
> > >> new
> > >> > > and
> > >> > > > > > > unified
> > >> > > > > > > > > > > delimiter works.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I hope this explains the problem, the solution,
> and
> > >> that
> > >> > > this
> > >> > > > > can
> > >> > > > > > > > make
> > >> > > > > > > > > it
> > >> > > > > > > > > > > in the next 0.8.x.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Otis
> > >> > > > > > > > > > > --
> > >> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> > Centralized
> > >> > Log
> > >> > > > > > > > Management
> > >> > > > > > > > > > > Solr & Elasticsearch Support *
> http://sematext.com/
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>



-- 
Thanks,
Raja.

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Neha Narkhede <ne...@gmail.com>.

+1 on getting rid of the quotes.

On Fri, Oct 17, 2014 at 12:31 PM, Magnus Spångdal <
magnus.spangdal@deltaprojects.com> wrote:

> +1 to get rid of quotes, thanks!
>
>
>
>
>
>
> —
> Sent from Mailbox
>
> On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Hi, everyone,
> > We are fixing the mbean names in kafka-1482, by adding separate explicit
> > tags in the name for things like clientId and topic. Another thing that
> > some people have complained before is that we use quotes in the jmx name.
> > Should we also just get rid of the quotes as part of kafka-1482? So,
> > instead of
> >    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > we will have
> >    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> > Thanks,
> > Jun
> > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <ne...@gmail.com>
> > wrote:
> >> I am going to vote for 1482 to be included in 0.8.2, if we have a patch
> >> submitted in a week. I think we've had this JIRA opened for too long
> and we
> >> held people back so it's only fair to release this.
> >>
> >> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
> >>
> >> > Otis,
> >> >
> >> > Just have the patch ready asap. We can make a call then.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> >> > otis.gospodnetic@gmail.com
> >> > > wrote:
> >> >
> >> > > Hi Jun,
> >> > >
> >> > > Would by the end of next week be acceptable for 0.8.2?
> >> > >
> >> > > Thanks,
> >> > > Otis
> >> > > --
> >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> >> > > Solr & Elasticsearch Support * http://sematext.com/
> >> > >
> >> > >
> >> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> >> > >
> >> > > > Otis,
> >> > > >
> >> > > > Yes, if you guys can help provide a patch in a few days, we can
> >> > probably
> >> > > > get it to the 0.8.2 release.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> >> > > > otis.gospodnetic@gmail.com> wrote:
> >> > > >
> >> > > > > Hi Jun,
> >> > > > >
> >> > > > > I think your MBean renaming approach will work.  I see
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix
> Version
> >> > > 0.8.2,
> >> > > > > but
> >> > > > > is not marked as a Blocker.  We'd love to get the MBeans fixed
> so
> >> > this
> >> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
> >> plate
> >> > > (the
> >> > > > > issue is currently Unassigned)?  If not, should we provide a new
> >> > patch
> >> > > > that
> >> > > > > uses your approach?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Otis
> >> > > > > --
> >> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> > Management
> >> > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
> wrote:
> >> > > > >
> >> > > > > > Otis,
> >> > > > > >
> >> > > > > > In kafka-1481, we will have to change the mbean names (at
> least
> >> the
> >> > > > ones
> >> > > > > > with clientid and topic) anyway. Using the name/value pair in
> the
> >> > > mbean
> >> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
> >> allowed
> >> > > in
> >> > > > > > clientid or topic.
> >> > > > > >
> >> > > > > > Bhavesh,
> >> > > > > >
> >> > > > > > Yes, I was thinking of making changes in the new metrics
> package.
> >> > > > > Something
> >> > > > > > like allowing the sensor names to have name/value pairs. The
> jmx
> >> > > names
> >> > > > > will
> >> > > > > > just follow accordingly. This is probably cleaner than doing
> the
> >> > > > > escaping.
> >> > > > > > Also, the metric names are more intuitive (otherwise, you
> have to
> >> > > know
> >> > > > > > which part is the clientid and which part is the topic).
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > Jun
> >> > > > > >
> >> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> >> > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > >
> >> > > > > > > Hi Jun,
> >> > > > > > >
> >> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <junrao@gmail.com
> >
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > Bhavesh,
> >> > > > > > > >
> >> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit
> harder
> >> > to
> >> > > > > define
> >> > > > > > > the
> >> > > > > > > > JMX bean names. I see a couple of solutions here.
> >> > > > > > > >
> >> > > > > > > > 1. Disable dot in clientId and topic names. The issue is
> that
> >> > dot
> >> > > > may
> >> > > > > > > > already be used in existing deployment.
> >> > > > > > > >
> >> > > > > > > > 2. We can represent the JMX bean name differently in the
> new
> >> > > > > producer.
> >> > > > > > > > Instead of
> >> > > > > > > >   kafka.producer.myclientid:type=mytopic
> >> > > > > > > > we could change it to
> >> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> >> > > > > > > >
> >> > > > > > > > I felt that option 2 is probably better since it doesn't
> >> affect
> >> > > > > > existing
> >> > > > > > > > users.
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > If it doesn't affect existing users, great.
> >> > > > > > >
> >> > > > > > > If you are saying that each "piece" of MBean name could be
> >> > > expressed
> >> > > > as
> >> > > > > > > name=value pair, with something like "," (forbidden in host
> >> > names,
> >> > > > > topic
> >> > > > > > > names, client IDs, etc. I assume?) then yes, I think this
> would
> >> > be
> >> > > > > easier
> >> > > > > > > to parse and it would be easier for people to understand
> what
> >> is
> >> > > > what.
> >> > > > > > >
> >> > > > > > > Otis
> >> > > > > > > --
> >> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> > > > Management
> >> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > Otis,
> >> > > > > > > >
> >> > > > > > > > We probably can also use option 2 to address KAFKA-1481.
> For
> >> > > > > > > topic/clientid
> >> > > > > > > > specific metrics, we could explicitly specify the metric
> name
> >> > so
> >> > > > that
> >> > > > > > it
> >> > > > > > > > contains "topic=mytopic,clientid=myclientid". That seems
> to
> >> be
> >> > a
> >> > > > much
> >> > > > > > > > cleaner way than having all parts included in a single
> string
> >> > > > > separated
> >> > > > > > > by
> >> > > > > > > > '|'.
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > >
> >> > > > > > > > Jun
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> >> > > > > > > > mistry.p.bhavesh@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > HI Otis,
> >> > > > > > > > >
> >> > > > > > > > > What is migration path ?  If topic with special chars
> >> exists
> >> > > > > already(
> >> > > > > > > > > ".","-","|" etc)  in previous version of
> producer/consumer
> >> of
> >> > > > > Kafka,
> >> > > > > > > what
> >> > > > > > > > > happens after the upgrade new producer or consumer
> (kafka
> >> > > > version)
> >> > > > > ?
> >> > > > > > > > Also,
> >> > > > > > > > > in new producer API (Kafka Trunk), does this enforce the
> >> rule
> >> > > > about
> >> > > > > > > > client
> >> > > > > > > > > id as well ?
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Bhavesh
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> >> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > > So maybe I should I should have asked the Q
> explicitly:
> >> > > > > > > > > > Could we commit the patch from
> >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now
> >> > that, I
> >> > > > > hope,
> >> > > > > > > > it's
> >> > > > > > > > > > clear what problems the current MBean names can cause?
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Otis
> >> > > > > > > > > > --
> >> > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> Centralized
> >> Log
> >> > > > > > > Management
> >> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> >> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi,
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Problem:*
> >> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of
> things
> >> > like
> >> > > > > > > <consumer
> >> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are
> used
> >> > as
> >> > > > > > > > delimiters.
> >> > > > > > > > > > >  When <consumer group> and <topic> don't contain the
> >> > > > delimiter
> >> > > > > > > > > character
> >> > > > > > > > > > > all is good if you want to extract parts of this
> MBean
> >> > name
> >> > > > by
> >> > > > > > > simply
> >> > > > > > > > > > > splitting on the delimiter character.  The problem
> is
> >> > that
> >> > > > > dashes
> >> > > > > > > are
> >> > > > > > > > > > > allowed in topic and group names, so this splitting
> >> > doesn't
> >> > > > > work.
> >> > > > > > > > > > > Moreover, underscores are also used as delimiters,
> and
> >> > they
> >> > > > can
> >> > > > > > > also
> >> > > > > > > > be
> >> > > > > > > > > > > used in things like topic names.
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Example*:
> >> > > > > > > > > > > This MBean's name is composed of <consumer
> >> > > > > > > > group>-<topic>-BytesPerSec:
> >> > > > > > > > > > >
> >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> >> > > > > > > > > name="*myGroup**-myTopic**-*
> >> > > > > > > > > > > BytesPerSec"
> >> > > > > > > > > > >
> >> > > > > > > > > > > Here we can actually split on "-" and extract all 3
> >> parts
> >> > > > from
> >> > > > > > the
> >> > > > > > > > > MBean
> >> > > > > > > > > > > name::
> >> > > > > > > > > > > * consumer group ('*myGroup*')
> >> > > > > > > > > > > * topic ('*myTopic*')
> >> > > > > > > > > > > * metric (‘BytesPerSec’)
> >> > > > > > > > > > >
> >> > > > > > > > > > > All good!
> >> > > > > > > > > > >
> >> > > > > > > > > > > But imagine if I named the group: *my-Group*
> >> > > > > > > > > > > And if I named the topic: *my-Topic*
> >> > > > > > > > > > >
> >> > > > > > > > > > > Then we'd have:
> >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> >> > > > > > > > > > name="*my-Group**-my-Topic**-*
> >> > > > > > > > > > > BytesPerSec"
> >> > > > > > > > > > >
> >> > > > > > > > > > > Now splitting on "-" would no longer work!  To
> extract
> >> > > > > "my-Group"
> >> > > > > > > and
> >> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to
> know
> >> > the
> >> > > > > > > specific
> >> > > > > > > > > > group
> >> > > > > > > > > > > name and topic name to look for and could not use
> >> generic
> >> > > > > > approach
> >> > > > > > > of
> >> > > > > > > > > > just
> >> > > > > > > > > > > splitting the MBean name on the delimiter.
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Solution*:
> >> > > > > > > > > > > The patch in
> >> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> >> > > > > > > > replaces
> >> > > > > > > > > > > all _ and - characters where they are used as
> >> delimiters
> >> > in
> >> > > > > MBean
> >> > > > > > > > names
> >> > > > > > > > > > > with a "|" character.  Because the "I" character is
> not
> >> > > > allowed
> >> > > > > > in
> >> > > > > > > > > topic
> >> > > > > > > > > > > names, consumer groups, host names, splitting on
> this
> >> new
> >> > > and
> >> > > > > > > unified
> >> > > > > > > > > > > delimiter works.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I hope this explains the problem, the solution, and
> >> that
> >> > > this
> >> > > > > can
> >> > > > > > > > make
> >> > > > > > > > > it
> >> > > > > > > > > > > in the next 0.8.x.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Otis
> >> > > > > > > > > > > --
> >> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> Centralized
> >> > Log
> >> > > > > > > > Management
> >> > > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Magnus Spångdal <ma...@deltaprojects.com>.

+1 to get rid of quotes, thanks!






—
Sent from Mailbox

On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao <ju...@gmail.com> wrote:

> Hi, everyone,
> We are fixing the mbean names in kafka-1482, by adding separate explicit
> tags in the name for things like clientId and topic. Another thing that
> some people have complained before is that we use quotes in the jmx name.
> Should we also just get rid of the quotes as part of kafka-1482? So,
> instead of
>    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> we will have
>    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> Thanks,
> Jun
> On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <ne...@gmail.com>
> wrote:
>> I am going to vote for 1482 to be included in 0.8.2, if we have a patch
>> submitted in a week. I think we've had this JIRA opened for too long and we
>> held people back so it's only fair to release this.
>>
>> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
>>
>> > Otis,
>> >
>> > Just have the patch ready asap. We can make a call then.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
>> > otis.gospodnetic@gmail.com
>> > > wrote:
>> >
>> > > Hi Jun,
>> > >
>> > > Would by the end of next week be acceptable for 0.8.2?
>> > >
>> > > Thanks,
>> > > Otis
>> > > --
>> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> > > Solr & Elasticsearch Support * http://sematext.com/
>> > >
>> > >
>> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
>> > >
>> > > > Otis,
>> > > >
>> > > > Yes, if you guys can help provide a patch in a few days, we can
>> > probably
>> > > > get it to the 0.8.2 release.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jun
>> > > >
>> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
>> > > > otis.gospodnetic@gmail.com> wrote:
>> > > >
>> > > > > Hi Jun,
>> > > > >
>> > > > > I think your MBean renaming approach will work.  I see
>> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version
>> > > 0.8.2,
>> > > > > but
>> > > > > is not marked as a Blocker.  We'd love to get the MBeans fixed so
>> > this
>> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
>> plate
>> > > (the
>> > > > > issue is currently Unassigned)?  If not, should we provide a new
>> > patch
>> > > > that
>> > > > > uses your approach?
>> > > > >
>> > > > > Thanks,
>> > > > > Otis
>> > > > > --
>> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> > Management
>> > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > >
>> > > > >
>> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com> wrote:
>> > > > >
>> > > > > > Otis,
>> > > > > >
>> > > > > > In kafka-1481, we will have to change the mbean names (at least
>> the
>> > > > ones
>> > > > > > with clientid and topic) anyway. Using the name/value pair in the
>> > > mbean
>> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
>> allowed
>> > > in
>> > > > > > clientid or topic.
>> > > > > >
>> > > > > > Bhavesh,
>> > > > > >
>> > > > > > Yes, I was thinking of making changes in the new metrics package.
>> > > > > Something
>> > > > > > like allowing the sensor names to have name/value pairs. The jmx
>> > > names
>> > > > > will
>> > > > > > just follow accordingly. This is probably cleaner than doing the
>> > > > > escaping.
>> > > > > > Also, the metric names are more intuitive (otherwise, you have to
>> > > know
>> > > > > > which part is the clientid and which part is the topic).
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Jun
>> > > > > >
>> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
>> > > > > > otis.gospodnetic@gmail.com> wrote:
>> > > > > >
>> > > > > > > Hi Jun,
>> > > > > > >
>> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com>
>> > > wrote:
>> > > > > > >
>> > > > > > > > Bhavesh,
>> > > > > > > >
>> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit harder
>> > to
>> > > > > define
>> > > > > > > the
>> > > > > > > > JMX bean names. I see a couple of solutions here.
>> > > > > > > >
>> > > > > > > > 1. Disable dot in clientId and topic names. The issue is that
>> > dot
>> > > > may
>> > > > > > > > already be used in existing deployment.
>> > > > > > > >
>> > > > > > > > 2. We can represent the JMX bean name differently in the new
>> > > > > producer.
>> > > > > > > > Instead of
>> > > > > > > >   kafka.producer.myclientid:type=mytopic
>> > > > > > > > we could change it to
>> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
>> > > > > > > >
>> > > > > > > > I felt that option 2 is probably better since it doesn't
>> affect
>> > > > > > existing
>> > > > > > > > users.
>> > > > > > > >
>> > > > > > >
>> > > > > > > If it doesn't affect existing users, great.
>> > > > > > >
>> > > > > > > If you are saying that each "piece" of MBean name could be
>> > > expressed
>> > > > as
>> > > > > > > name=value pair, with something like "," (forbidden in host
>> > names,
>> > > > > topic
>> > > > > > > names, client IDs, etc. I assume?) then yes, I think this would
>> > be
>> > > > > easier
>> > > > > > > to parse and it would be easier for people to understand what
>> is
>> > > > what.
>> > > > > > >
>> > > > > > > Otis
>> > > > > > > --
>> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> > > > Management
>> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > >
>> > > > > > > > Otis,
>> > > > > > > >
>> > > > > > > > We probably can also use option 2 to address KAFKA-1481. For
>> > > > > > > topic/clientid
>> > > > > > > > specific metrics, we could explicitly specify the metric name
>> > so
>> > > > that
>> > > > > > it
>> > > > > > > > contains "topic=mytopic,clientid=myclientid". That seems to
>> be
>> > a
>> > > > much
>> > > > > > > > cleaner way than having all parts included in a single string
>> > > > > separated
>> > > > > > > by
>> > > > > > > > '|'.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > >
>> > > > > > > > Jun
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
>> > > > > > > > mistry.p.bhavesh@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > HI Otis,
>> > > > > > > > >
>> > > > > > > > > What is migration path ?  If topic with special chars
>> exists
>> > > > > already(
>> > > > > > > > > ".","-","|" etc)  in previous version of producer/consumer
>> of
>> > > > > Kafka,
>> > > > > > > what
>> > > > > > > > > happens after the upgrade new producer or consumer (kafka
>> > > > version)
>> > > > > ?
>> > > > > > > > Also,
>> > > > > > > > > in new producer API (Kafka Trunk), does this enforce the
>> rule
>> > > > about
>> > > > > > > > client
>> > > > > > > > > id as well ?
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Bhavesh
>> > > > > > > > >
>> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
>> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi,
>> > > > > > > > > >
>> > > > > > > > > > So maybe I should I should have asked the Q explicitly:
>> > > > > > > > > > Could we commit the patch from
>> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now
>> > that, I
>> > > > > hope,
>> > > > > > > > it's
>> > > > > > > > > > clear what problems the current MBean names can cause?
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Otis
>> > > > > > > > > > --
>> > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
>> Log
>> > > > > > > Management
>> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
>> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi,
>> > > > > > > > > > >
>> > > > > > > > > > > *Problem:*
>> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of things
>> > like
>> > > > > > > <consumer
>> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are used
>> > as
>> > > > > > > > delimiters.
>> > > > > > > > > > >  When <consumer group> and <topic> don't contain the
>> > > > delimiter
>> > > > > > > > > character
>> > > > > > > > > > > all is good if you want to extract parts of this MBean
>> > name
>> > > > by
>> > > > > > > simply
>> > > > > > > > > > > splitting on the delimiter character.  The problem is
>> > that
>> > > > > dashes
>> > > > > > > are
>> > > > > > > > > > > allowed in topic and group names, so this splitting
>> > doesn't
>> > > > > work.
>> > > > > > > > > > > Moreover, underscores are also used as delimiters, and
>> > they
>> > > > can
>> > > > > > > also
>> > > > > > > > be
>> > > > > > > > > > > used in things like topic names.
>> > > > > > > > > > >
>> > > > > > > > > > > *Example*:
>> > > > > > > > > > > This MBean's name is composed of <consumer
>> > > > > > > > group>-<topic>-BytesPerSec:
>> > > > > > > > > > >
>> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
>> > > > > > > > > name="*myGroup**-myTopic**-*
>> > > > > > > > > > > BytesPerSec"
>> > > > > > > > > > >
>> > > > > > > > > > > Here we can actually split on "-" and extract all 3
>> parts
>> > > > from
>> > > > > > the
>> > > > > > > > > MBean
>> > > > > > > > > > > name::
>> > > > > > > > > > > * consumer group ('*myGroup*')
>> > > > > > > > > > > * topic ('*myTopic*')
>> > > > > > > > > > > * metric (‘BytesPerSec’)
>> > > > > > > > > > >
>> > > > > > > > > > > All good!
>> > > > > > > > > > >
>> > > > > > > > > > > But imagine if I named the group: *my-Group*
>> > > > > > > > > > > And if I named the topic: *my-Topic*
>> > > > > > > > > > >
>> > > > > > > > > > > Then we'd have:
>> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
>> > > > > > > > > > name="*my-Group**-my-Topic**-*
>> > > > > > > > > > > BytesPerSec"
>> > > > > > > > > > >
>> > > > > > > > > > > Now splitting on "-" would no longer work!  To extract
>> > > > > "my-Group"
>> > > > > > > and
>> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to know
>> > the
>> > > > > > > specific
>> > > > > > > > > > group
>> > > > > > > > > > > name and topic name to look for and could not use
>> generic
>> > > > > > approach
>> > > > > > > of
>> > > > > > > > > > just
>> > > > > > > > > > > splitting the MBean name on the delimiter.
>> > > > > > > > > > >
>> > > > > > > > > > > *Solution*:
>> > > > > > > > > > > The patch in
>> > > > https://issues.apache.org/jira/browse/KAFKA-1481
>> > > > > > > > replaces
>> > > > > > > > > > > all _ and - characters where they are used as
>> delimiters
>> > in
>> > > > > MBean
>> > > > > > > > names
>> > > > > > > > > > > with a "|" character.  Because the "I" character is not
>> > > > allowed
>> > > > > > in
>> > > > > > > > > topic
>> > > > > > > > > > > names, consumer groups, host names, splitting on this
>> new
>> > > and
>> > > > > > > unified
>> > > > > > > > > > > delimiter works.
>> > > > > > > > > > >
>> > > > > > > > > > > I hope this explains the problem, the solution, and
>> that
>> > > this
>> > > > > can
>> > > > > > > > make
>> > > > > > > > > it
>> > > > > > > > > > > in the next 0.8.x.
>> > > > > > > > > > >
>> > > > > > > > > > > Otis
>> > > > > > > > > > > --
>> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
>> > Log
>> > > > > > > > Management
>> > > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Otis Gospodnetic <ot...@gmail.com>.

Note: https://issues.apache.org/jira/browse/KAFKA-1481 (not 1482, just so
there is no confusion)

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Fri, Oct 31, 2014 at 2:10 PM, Jun Rao <ju...@gmail.com> wrote:

> Yes, all changes are related to metric names. Feel free to review the
> patch.
>
> Thanks,
>
> Jun
>
> On Fri, Oct 31, 2014 at 10:48 AM, Joel Koshy <jj...@gmail.com> wrote:
>
> > That sounds good, although is that the only change (sorry I have not
> > done a careful review of that patch and would like to before it gets
> > checked in).
> >
> > On Fri, Oct 31, 2014 at 10:42:13AM -0700, Jun Rao wrote:
> > > To circle back on this thread. The patch in kafka-1482 is almost ready.
> > To
> > > make the mbean names more meaningful and easier to parse, the patch
> will
> > > use explicit key/value pairs in the mbean name for things like clientId
> > and
> > > topic, and will get rid of the quotes.
> > >
> > > So, instead of
> > >
> "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > > we will have
> > >
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> > >
> > > Any objection to committing this to the 0.8.2 branch?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Oct 17, 2014 at 11:54 AM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Hi, everyone,
> > > >
> > > > We are fixing the mbean names in kafka-1482, by adding separate
> > explicit
> > > > tags in the name for things like clientId and topic. Another thing
> that
> > > > some people have complained before is that we use quotes in the jmx
> > name.
> > > > Should we also just get rid of the quotes as part of kafka-1482? So,
> > > > instead of
> > > >
> > "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > > > we will have
> > > >
> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <
> > neha.narkhede@gmail.com>
> > > > wrote:
> > > >
> > > >> I am going to vote for 1482 to be included in 0.8.2, if we have a
> > patch
> > > >> submitted in a week. I think we've had this JIRA opened for too long
> > and
> > > >> we
> > > >> held people back so it's only fair to release this.
> > > >>
> > > >> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
> > > >>
> > > >> > Otis,
> > > >> >
> > > >> > Just have the patch ready asap. We can make a call then.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Jun
> > > >> >
> > > >> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> > > >> > otis.gospodnetic@gmail.com
> > > >> > > wrote:
> > > >> >
> > > >> > > Hi Jun,
> > > >> > >
> > > >> > > Would by the end of next week be acceptable for 0.8.2?
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Otis
> > > >> > > --
> > > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > > >> > > Solr & Elasticsearch Support * http://sematext.com/
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com>
> wrote:
> > > >> > >
> > > >> > > > Otis,
> > > >> > > >
> > > >> > > > Yes, if you guys can help provide a patch in a few days, we
> can
> > > >> > probably
> > > >> > > > get it to the 0.8.2 release.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > >
> > > >> > > > Jun
> > > >> > > >
> > > >> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> > > >> > > > otis.gospodnetic@gmail.com> wrote:
> > > >> > > >
> > > >> > > > > Hi Jun,
> > > >> > > > >
> > > >> > > > > I think your MBean renaming approach will work.  I see
> > > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix
> > Version
> > > >> > > 0.8.2,
> > > >> > > > > but
> > > >> > > > > is not marked as a Blocker.  We'd love to get the MBeans
> > fixed so
> > > >> > this
> > > >> > > > > makes it in 0.8.2 release.  Do you know if this is on
> anyone's
> > > >> plate
> > > >> > > (the
> > > >> > > > > issue is currently Unassigned)?  If not, should we provide a
> > new
> > > >> > patch
> > > >> > > > that
> > > >> > > > > uses your approach?
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Otis
> > > >> > > > > --
> > > >> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > >> > Management
> > > >> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
> > > >> wrote:
> > > >> > > > >
> > > >> > > > > > Otis,
> > > >> > > > > >
> > > >> > > > > > In kafka-1481, we will have to change the mbean names (at
> > least
> > > >> the
> > > >> > > > ones
> > > >> > > > > > with clientid and topic) anyway. Using the name/value pair
> > in
> > > >> the
> > > >> > > mbean
> > > >> > > > > > name allows us to do this in a cleaner way. Yes, "," is
> not
> > > >> allowed
> > > >> > > in
> > > >> > > > > > clientid or topic.
> > > >> > > > > >
> > > >> > > > > > Bhavesh,
> > > >> > > > > >
> > > >> > > > > > Yes, I was thinking of making changes in the new metrics
> > > >> package.
> > > >> > > > > Something
> > > >> > > > > > like allowing the sensor names to have name/value pairs.
> > The jmx
> > > >> > > names
> > > >> > > > > will
> > > >> > > > > > just follow accordingly. This is probably cleaner than
> > doing the
> > > >> > > > > escaping.
> > > >> > > > > > Also, the metric names are more intuitive (otherwise, you
> > have
> > > >> to
> > > >> > > know
> > > >> > > > > > which part is the clientid and which part is the topic).
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > >
> > > >> > > > > > Jun
> > > >> > > > > >
> > > >> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > > >> > > > > > otis.gospodnetic@gmail.com> wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Jun,
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <
> > junrao@gmail.com>
> > > >> > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Bhavesh,
> > > >> > > > > > > >
> > > >> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit
> > > >> harder
> > > >> > to
> > > >> > > > > define
> > > >> > > > > > > the
> > > >> > > > > > > > JMX bean names. I see a couple of solutions here.
> > > >> > > > > > > >
> > > >> > > > > > > > 1. Disable dot in clientId and topic names. The issue
> is
> > > >> that
> > > >> > dot
> > > >> > > > may
> > > >> > > > > > > > already be used in existing deployment.
> > > >> > > > > > > >
> > > >> > > > > > > > 2. We can represent the JMX bean name differently in
> > the new
> > > >> > > > > producer.
> > > >> > > > > > > > Instead of
> > > >> > > > > > > >   kafka.producer.myclientid:type=mytopic
> > > >> > > > > > > > we could change it to
> > > >> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > > >> > > > > > > >
> > > >> > > > > > > > I felt that option 2 is probably better since it
> doesn't
> > > >> affect
> > > >> > > > > > existing
> > > >> > > > > > > > users.
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > If it doesn't affect existing users, great.
> > > >> > > > > > >
> > > >> > > > > > > If you are saying that each "piece" of MBean name could
> be
> > > >> > > expressed
> > > >> > > > as
> > > >> > > > > > > name=value pair, with something like "," (forbidden in
> > host
> > > >> > names,
> > > >> > > > > topic
> > > >> > > > > > > names, client IDs, etc. I assume?) then yes, I think
> this
> > > >> would
> > > >> > be
> > > >> > > > > easier
> > > >> > > > > > > to parse and it would be easier for people to understand
> > what
> > > >> is
> > > >> > > > what.
> > > >> > > > > > >
> > > >> > > > > > > Otis
> > > >> > > > > > > --
> > > >> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> > Log
> > > >> > > > Management
> > > >> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > Otis,
> > > >> > > > > > > >
> > > >> > > > > > > > We probably can also use option 2 to address
> > KAFKA-1481. For
> > > >> > > > > > > topic/clientid
> > > >> > > > > > > > specific metrics, we could explicitly specify the
> metric
> > > >> name
> > > >> > so
> > > >> > > > that
> > > >> > > > > > it
> > > >> > > > > > > > contains "topic=mytopic,clientid=myclientid". That
> > seems to
> > > >> be
> > > >> > a
> > > >> > > > much
> > > >> > > > > > > > cleaner way than having all parts included in a single
> > > >> string
> > > >> > > > > separated
> > > >> > > > > > > by
> > > >> > > > > > > > '|'.
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > >
> > > >> > > > > > > > Jun
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > > >> > > > > > > > mistry.p.bhavesh@gmail.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > HI Otis,
> > > >> > > > > > > > >
> > > >> > > > > > > > > What is migration path ?  If topic with special
> chars
> > > >> exists
> > > >> > > > > already(
> > > >> > > > > > > > > ".","-","|" etc)  in previous version of
> > > >> producer/consumer of
> > > >> > > > > Kafka,
> > > >> > > > > > > what
> > > >> > > > > > > > > happens after the upgrade new producer or consumer
> > (kafka
> > > >> > > > version)
> > > >> > > > > ?
> > > >> > > > > > > > Also,
> > > >> > > > > > > > > in new producer API (Kafka Trunk), does this enforce
> > the
> > > >> rule
> > > >> > > > about
> > > >> > > > > > > > client
> > > >> > > > > > > > > id as well ?
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Bhavesh
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > > >> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Hi,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > So maybe I should I should have asked the Q
> > explicitly:
> > > >> > > > > > > > > > Could we commit the patch from
> > > >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481
> > now
> > > >> > that, I
> > > >> > > > > hope,
> > > >> > > > > > > > it's
> > > >> > > > > > > > > > clear what problems the current MBean names can
> > cause?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks,
> > > >> > > > > > > > > > Otis
> > > >> > > > > > > > > > --
> > > >> > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> > Centralized
> > > >> Log
> > > >> > > > > > > Management
> > > >> > > > > > > > > > Solr & Elasticsearch Support *
> http://sematext.com/
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis
> Gospodnetic <
> > > >> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > *Problem:*
> > > >> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of
> > things
> > > >> > like
> > > >> > > > > > > <consumer
> > > >> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes
> are
> > > >> used
> > > >> > as
> > > >> > > > > > > > delimiters.
> > > >> > > > > > > > > > >  When <consumer group> and <topic> don't contain
> > the
> > > >> > > > delimiter
> > > >> > > > > > > > > character
> > > >> > > > > > > > > > > all is good if you want to extract parts of this
> > MBean
> > > >> > name
> > > >> > > > by
> > > >> > > > > > > simply
> > > >> > > > > > > > > > > splitting on the delimiter character.  The
> > problem is
> > > >> > that
> > > >> > > > > dashes
> > > >> > > > > > > are
> > > >> > > > > > > > > > > allowed in topic and group names, so this
> > splitting
> > > >> > doesn't
> > > >> > > > > work.
> > > >> > > > > > > > > > > Moreover, underscores are also used as
> > delimiters, and
> > > >> > they
> > > >> > > > can
> > > >> > > > > > > also
> > > >> > > > > > > > be
> > > >> > > > > > > > > > > used in things like topic names.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > *Example*:
> > > >> > > > > > > > > > > This MBean's name is composed of <consumer
> > > >> > > > > > > > group>-<topic>-BytesPerSec:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > >> > > > > > > > > name="*myGroup**-myTopic**-*
> > > >> > > > > > > > > > > BytesPerSec"
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Here we can actually split on "-" and extract
> all
> > 3
> > > >> parts
> > > >> > > > from
> > > >> > > > > > the
> > > >> > > > > > > > > MBean
> > > >> > > > > > > > > > > name::
> > > >> > > > > > > > > > > * consumer group ('*myGroup*')
> > > >> > > > > > > > > > > * topic ('*myTopic*')
> > > >> > > > > > > > > > > * metric (‘BytesPerSec’)
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > All good!
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > But imagine if I named the group: *my-Group*
> > > >> > > > > > > > > > > And if I named the topic: *my-Topic*
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Then we'd have:
> > > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > >> > > > > > > > > > name="*my-Group**-my-Topic**-*
> > > >> > > > > > > > > > > BytesPerSec"
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Now splitting on "-" would no longer work!  To
> > extract
> > > >> > > > > "my-Group"
> > > >> > > > > > > and
> > > >> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have
> to
> > > >> know
> > > >> > the
> > > >> > > > > > > specific
> > > >> > > > > > > > > > group
> > > >> > > > > > > > > > > name and topic name to look for and could not
> use
> > > >> generic
> > > >> > > > > > approach
> > > >> > > > > > > of
> > > >> > > > > > > > > > just
> > > >> > > > > > > > > > > splitting the MBean name on the delimiter.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > *Solution*:
> > > >> > > > > > > > > > > The patch in
> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> > > >> > > > > > > > replaces
> > > >> > > > > > > > > > > all _ and - characters where they are used as
> > > >> delimiters
> > > >> > in
> > > >> > > > > MBean
> > > >> > > > > > > > names
> > > >> > > > > > > > > > > with a "|" character.  Because the "I" character
> > is
> > > >> not
> > > >> > > > allowed
> > > >> > > > > > in
> > > >> > > > > > > > > topic
> > > >> > > > > > > > > > > names, consumer groups, host names, splitting on
> > this
> > > >> new
> > > >> > > and
> > > >> > > > > > > unified
> > > >> > > > > > > > > > > delimiter works.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I hope this explains the problem, the solution,
> > and
> > > >> that
> > > >> > > this
> > > >> > > > > can
> > > >> > > > > > > > make
> > > >> > > > > > > > > it
> > > >> > > > > > > > > > > in the next 0.8.x.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Otis
> > > >> > > > > > > > > > > --
> > > >> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> > > >> Centralized
> > > >> > Log
> > > >> > > > > > > > Management
> > > >> > > > > > > > > > > Solr & Elasticsearch Support *
> > http://sematext.com/
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Yes, all changes are related to metric names. Feel free to review the patch.

Thanks,

Jun

On Fri, Oct 31, 2014 at 10:48 AM, Joel Koshy <jj...@gmail.com> wrote:

> That sounds good, although is that the only change (sorry I have not
> done a careful review of that patch and would like to before it gets
> checked in).
>
> On Fri, Oct 31, 2014 at 10:42:13AM -0700, Jun Rao wrote:
> > To circle back on this thread. The patch in kafka-1482 is almost ready.
> To
> > make the mbean names more meaningful and easier to parse, the patch will
> > use explicit key/value pairs in the mbean name for things like clientId
> and
> > topic, and will get rid of the quotes.
> >
> > So, instead of
> >    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > we will have
> >    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> >
> > Any objection to committing this to the 0.8.2 branch?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Oct 17, 2014 at 11:54 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Hi, everyone,
> > >
> > > We are fixing the mbean names in kafka-1482, by adding separate
> explicit
> > > tags in the name for things like clientId and topic. Another thing that
> > > some people have complained before is that we use quotes in the jmx
> name.
> > > Should we also just get rid of the quotes as part of kafka-1482? So,
> > > instead of
> > >
> "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > > we will have
> > >
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <
> neha.narkhede@gmail.com>
> > > wrote:
> > >
> > >> I am going to vote for 1482 to be included in 0.8.2, if we have a
> patch
> > >> submitted in a week. I think we've had this JIRA opened for too long
> and
> > >> we
> > >> held people back so it's only fair to release this.
> > >>
> > >> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
> > >>
> > >> > Otis,
> > >> >
> > >> > Just have the patch ready asap. We can make a call then.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> > >> > otis.gospodnetic@gmail.com
> > >> > > wrote:
> > >> >
> > >> > > Hi Jun,
> > >> > >
> > >> > > Would by the end of next week be acceptable for 0.8.2?
> > >> > >
> > >> > > Thanks,
> > >> > > Otis
> > >> > > --
> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > >> > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > >
> > >> > >
> > >> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> > >> > >
> > >> > > > Otis,
> > >> > > >
> > >> > > > Yes, if you guys can help provide a patch in a few days, we can
> > >> > probably
> > >> > > > get it to the 0.8.2 release.
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> > >> > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > >
> > >> > > > > Hi Jun,
> > >> > > > >
> > >> > > > > I think your MBean renaming approach will work.  I see
> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix
> Version
> > >> > > 0.8.2,
> > >> > > > > but
> > >> > > > > is not marked as a Blocker.  We'd love to get the MBeans
> fixed so
> > >> > this
> > >> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
> > >> plate
> > >> > > (the
> > >> > > > > issue is currently Unassigned)?  If not, should we provide a
> new
> > >> > patch
> > >> > > > that
> > >> > > > > uses your approach?
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Otis
> > >> > > > > --
> > >> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > >> > Management
> > >> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > >
> > >> > > > >
> > >> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
> > >> wrote:
> > >> > > > >
> > >> > > > > > Otis,
> > >> > > > > >
> > >> > > > > > In kafka-1481, we will have to change the mbean names (at
> least
> > >> the
> > >> > > > ones
> > >> > > > > > with clientid and topic) anyway. Using the name/value pair
> in
> > >> the
> > >> > > mbean
> > >> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
> > >> allowed
> > >> > > in
> > >> > > > > > clientid or topic.
> > >> > > > > >
> > >> > > > > > Bhavesh,
> > >> > > > > >
> > >> > > > > > Yes, I was thinking of making changes in the new metrics
> > >> package.
> > >> > > > > Something
> > >> > > > > > like allowing the sensor names to have name/value pairs.
> The jmx
> > >> > > names
> > >> > > > > will
> > >> > > > > > just follow accordingly. This is probably cleaner than
> doing the
> > >> > > > > escaping.
> > >> > > > > > Also, the metric names are more intuitive (otherwise, you
> have
> > >> to
> > >> > > know
> > >> > > > > > which part is the clientid and which part is the topic).
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > Jun
> > >> > > > > >
> > >> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > >> > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > >
> > >> > > > > > > Hi Jun,
> > >> > > > > > >
> > >> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <
> junrao@gmail.com>
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Bhavesh,
> > >> > > > > > > >
> > >> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit
> > >> harder
> > >> > to
> > >> > > > > define
> > >> > > > > > > the
> > >> > > > > > > > JMX bean names. I see a couple of solutions here.
> > >> > > > > > > >
> > >> > > > > > > > 1. Disable dot in clientId and topic names. The issue is
> > >> that
> > >> > dot
> > >> > > > may
> > >> > > > > > > > already be used in existing deployment.
> > >> > > > > > > >
> > >> > > > > > > > 2. We can represent the JMX bean name differently in
> the new
> > >> > > > > producer.
> > >> > > > > > > > Instead of
> > >> > > > > > > >   kafka.producer.myclientid:type=mytopic
> > >> > > > > > > > we could change it to
> > >> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > >> > > > > > > >
> > >> > > > > > > > I felt that option 2 is probably better since it doesn't
> > >> affect
> > >> > > > > > existing
> > >> > > > > > > > users.
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > > If it doesn't affect existing users, great.
> > >> > > > > > >
> > >> > > > > > > If you are saying that each "piece" of MBean name could be
> > >> > > expressed
> > >> > > > as
> > >> > > > > > > name=value pair, with something like "," (forbidden in
> host
> > >> > names,
> > >> > > > > topic
> > >> > > > > > > names, client IDs, etc. I assume?) then yes, I think this
> > >> would
> > >> > be
> > >> > > > > easier
> > >> > > > > > > to parse and it would be easier for people to understand
> what
> > >> is
> > >> > > > what.
> > >> > > > > > >
> > >> > > > > > > Otis
> > >> > > > > > > --
> > >> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> Log
> > >> > > > Management
> > >> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Otis,
> > >> > > > > > > >
> > >> > > > > > > > We probably can also use option 2 to address
> KAFKA-1481. For
> > >> > > > > > > topic/clientid
> > >> > > > > > > > specific metrics, we could explicitly specify the metric
> > >> name
> > >> > so
> > >> > > > that
> > >> > > > > > it
> > >> > > > > > > > contains "topic=mytopic,clientid=myclientid". That
> seems to
> > >> be
> > >> > a
> > >> > > > much
> > >> > > > > > > > cleaner way than having all parts included in a single
> > >> string
> > >> > > > > separated
> > >> > > > > > > by
> > >> > > > > > > > '|'.
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > >
> > >> > > > > > > > Jun
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > >> > > > > > > > mistry.p.bhavesh@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > HI Otis,
> > >> > > > > > > > >
> > >> > > > > > > > > What is migration path ?  If topic with special chars
> > >> exists
> > >> > > > > already(
> > >> > > > > > > > > ".","-","|" etc)  in previous version of
> > >> producer/consumer of
> > >> > > > > Kafka,
> > >> > > > > > > what
> > >> > > > > > > > > happens after the upgrade new producer or consumer
> (kafka
> > >> > > > version)
> > >> > > > > ?
> > >> > > > > > > > Also,
> > >> > > > > > > > > in new producer API (Kafka Trunk), does this enforce
> the
> > >> rule
> > >> > > > about
> > >> > > > > > > > client
> > >> > > > > > > > > id as well ?
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > >
> > >> > > > > > > > > Bhavesh
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > >> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi,
> > >> > > > > > > > > >
> > >> > > > > > > > > > So maybe I should I should have asked the Q
> explicitly:
> > >> > > > > > > > > > Could we commit the patch from
> > >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481
> now
> > >> > that, I
> > >> > > > > hope,
> > >> > > > > > > > it's
> > >> > > > > > > > > > clear what problems the current MBean names can
> cause?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Otis
> > >> > > > > > > > > > --
> > >> > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> Centralized
> > >> Log
> > >> > > > > > > Management
> > >> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > >> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Problem:*
> > >> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of
> things
> > >> > like
> > >> > > > > > > <consumer
> > >> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are
> > >> used
> > >> > as
> > >> > > > > > > > delimiters.
> > >> > > > > > > > > > >  When <consumer group> and <topic> don't contain
> the
> > >> > > > delimiter
> > >> > > > > > > > > character
> > >> > > > > > > > > > > all is good if you want to extract parts of this
> MBean
> > >> > name
> > >> > > > by
> > >> > > > > > > simply
> > >> > > > > > > > > > > splitting on the delimiter character.  The
> problem is
> > >> > that
> > >> > > > > dashes
> > >> > > > > > > are
> > >> > > > > > > > > > > allowed in topic and group names, so this
> splitting
> > >> > doesn't
> > >> > > > > work.
> > >> > > > > > > > > > > Moreover, underscores are also used as
> delimiters, and
> > >> > they
> > >> > > > can
> > >> > > > > > > also
> > >> > > > > > > > be
> > >> > > > > > > > > > > used in things like topic names.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Example*:
> > >> > > > > > > > > > > This MBean's name is composed of <consumer
> > >> > > > > > > > group>-<topic>-BytesPerSec:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > >> > > > > > > > > name="*myGroup**-myTopic**-*
> > >> > > > > > > > > > > BytesPerSec"
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Here we can actually split on "-" and extract all
> 3
> > >> parts
> > >> > > > from
> > >> > > > > > the
> > >> > > > > > > > > MBean
> > >> > > > > > > > > > > name::
> > >> > > > > > > > > > > * consumer group ('*myGroup*')
> > >> > > > > > > > > > > * topic ('*myTopic*')
> > >> > > > > > > > > > > * metric (‘BytesPerSec’)
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > All good!
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > But imagine if I named the group: *my-Group*
> > >> > > > > > > > > > > And if I named the topic: *my-Topic*
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Then we'd have:
> > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > >> > > > > > > > > > name="*my-Group**-my-Topic**-*
> > >> > > > > > > > > > > BytesPerSec"
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Now splitting on "-" would no longer work!  To
> extract
> > >> > > > > "my-Group"
> > >> > > > > > > and
> > >> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to
> > >> know
> > >> > the
> > >> > > > > > > specific
> > >> > > > > > > > > > group
> > >> > > > > > > > > > > name and topic name to look for and could not use
> > >> generic
> > >> > > > > > approach
> > >> > > > > > > of
> > >> > > > > > > > > > just
> > >> > > > > > > > > > > splitting the MBean name on the delimiter.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Solution*:
> > >> > > > > > > > > > > The patch in
> > >> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> > >> > > > > > > > replaces
> > >> > > > > > > > > > > all _ and - characters where they are used as
> > >> delimiters
> > >> > in
> > >> > > > > MBean
> > >> > > > > > > > names
> > >> > > > > > > > > > > with a "|" character.  Because the "I" character
> is
> > >> not
> > >> > > > allowed
> > >> > > > > > in
> > >> > > > > > > > > topic
> > >> > > > > > > > > > > names, consumer groups, host names, splitting on
> this
> > >> new
> > >> > > and
> > >> > > > > > > unified
> > >> > > > > > > > > > > delimiter works.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I hope this explains the problem, the solution,
> and
> > >> that
> > >> > > this
> > >> > > > > can
> > >> > > > > > > > make
> > >> > > > > > > > > it
> > >> > > > > > > > > > > in the next 0.8.x.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Otis
> > >> > > > > > > > > > > --
> > >> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> > >> Centralized
> > >> > Log
> > >> > > > > > > > Management
> > >> > > > > > > > > > > Solr & Elasticsearch Support *
> http://sematext.com/
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
>
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Yes, all changes are related to metric names. Feel free to review the patch.

Thanks,

Jun

On Fri, Oct 31, 2014 at 10:48 AM, Joel Koshy <jj...@gmail.com> wrote:

> That sounds good, although is that the only change (sorry I have not
> done a careful review of that patch and would like to before it gets
> checked in).
>
> On Fri, Oct 31, 2014 at 10:42:13AM -0700, Jun Rao wrote:
> > To circle back on this thread. The patch in kafka-1482 is almost ready.
> To
> > make the mbean names more meaningful and easier to parse, the patch will
> > use explicit key/value pairs in the mbean name for things like clientId
> and
> > topic, and will get rid of the quotes.
> >
> > So, instead of
> >    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > we will have
> >    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> >
> > Any objection to committing this to the 0.8.2 branch?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Oct 17, 2014 at 11:54 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Hi, everyone,
> > >
> > > We are fixing the mbean names in kafka-1482, by adding separate
> explicit
> > > tags in the name for things like clientId and topic. Another thing that
> > > some people have complained before is that we use quotes in the jmx
> name.
> > > Should we also just get rid of the quotes as part of kafka-1482? So,
> > > instead of
> > >
> "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > > we will have
> > >
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <
> neha.narkhede@gmail.com>
> > > wrote:
> > >
> > >> I am going to vote for 1482 to be included in 0.8.2, if we have a
> patch
> > >> submitted in a week. I think we've had this JIRA opened for too long
> and
> > >> we
> > >> held people back so it's only fair to release this.
> > >>
> > >> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
> > >>
> > >> > Otis,
> > >> >
> > >> > Just have the patch ready asap. We can make a call then.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> > >> > otis.gospodnetic@gmail.com
> > >> > > wrote:
> > >> >
> > >> > > Hi Jun,
> > >> > >
> > >> > > Would by the end of next week be acceptable for 0.8.2?
> > >> > >
> > >> > > Thanks,
> > >> > > Otis
> > >> > > --
> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > >> > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > >
> > >> > >
> > >> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> > >> > >
> > >> > > > Otis,
> > >> > > >
> > >> > > > Yes, if you guys can help provide a patch in a few days, we can
> > >> > probably
> > >> > > > get it to the 0.8.2 release.
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> > >> > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > >
> > >> > > > > Hi Jun,
> > >> > > > >
> > >> > > > > I think your MBean renaming approach will work.  I see
> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix
> Version
> > >> > > 0.8.2,
> > >> > > > > but
> > >> > > > > is not marked as a Blocker.  We'd love to get the MBeans
> fixed so
> > >> > this
> > >> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
> > >> plate
> > >> > > (the
> > >> > > > > issue is currently Unassigned)?  If not, should we provide a
> new
> > >> > patch
> > >> > > > that
> > >> > > > > uses your approach?
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Otis
> > >> > > > > --
> > >> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > >> > Management
> > >> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > >
> > >> > > > >
> > >> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
> > >> wrote:
> > >> > > > >
> > >> > > > > > Otis,
> > >> > > > > >
> > >> > > > > > In kafka-1481, we will have to change the mbean names (at
> least
> > >> the
> > >> > > > ones
> > >> > > > > > with clientid and topic) anyway. Using the name/value pair
> in
> > >> the
> > >> > > mbean
> > >> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
> > >> allowed
> > >> > > in
> > >> > > > > > clientid or topic.
> > >> > > > > >
> > >> > > > > > Bhavesh,
> > >> > > > > >
> > >> > > > > > Yes, I was thinking of making changes in the new metrics
> > >> package.
> > >> > > > > Something
> > >> > > > > > like allowing the sensor names to have name/value pairs.
> The jmx
> > >> > > names
> > >> > > > > will
> > >> > > > > > just follow accordingly. This is probably cleaner than
> doing the
> > >> > > > > escaping.
> > >> > > > > > Also, the metric names are more intuitive (otherwise, you
> have
> > >> to
> > >> > > know
> > >> > > > > > which part is the clientid and which part is the topic).
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > Jun
> > >> > > > > >
> > >> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > >> > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > >
> > >> > > > > > > Hi Jun,
> > >> > > > > > >
> > >> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <
> junrao@gmail.com>
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Bhavesh,
> > >> > > > > > > >
> > >> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit
> > >> harder
> > >> > to
> > >> > > > > define
> > >> > > > > > > the
> > >> > > > > > > > JMX bean names. I see a couple of solutions here.
> > >> > > > > > > >
> > >> > > > > > > > 1. Disable dot in clientId and topic names. The issue is
> > >> that
> > >> > dot
> > >> > > > may
> > >> > > > > > > > already be used in existing deployment.
> > >> > > > > > > >
> > >> > > > > > > > 2. We can represent the JMX bean name differently in
> the new
> > >> > > > > producer.
> > >> > > > > > > > Instead of
> > >> > > > > > > >   kafka.producer.myclientid:type=mytopic
> > >> > > > > > > > we could change it to
> > >> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > >> > > > > > > >
> > >> > > > > > > > I felt that option 2 is probably better since it doesn't
> > >> affect
> > >> > > > > > existing
> > >> > > > > > > > users.
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > > If it doesn't affect existing users, great.
> > >> > > > > > >
> > >> > > > > > > If you are saying that each "piece" of MBean name could be
> > >> > > expressed
> > >> > > > as
> > >> > > > > > > name=value pair, with something like "," (forbidden in
> host
> > >> > names,
> > >> > > > > topic
> > >> > > > > > > names, client IDs, etc. I assume?) then yes, I think this
> > >> would
> > >> > be
> > >> > > > > easier
> > >> > > > > > > to parse and it would be easier for people to understand
> what
> > >> is
> > >> > > > what.
> > >> > > > > > >
> > >> > > > > > > Otis
> > >> > > > > > > --
> > >> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> Log
> > >> > > > Management
> > >> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Otis,
> > >> > > > > > > >
> > >> > > > > > > > We probably can also use option 2 to address
> KAFKA-1481. For
> > >> > > > > > > topic/clientid
> > >> > > > > > > > specific metrics, we could explicitly specify the metric
> > >> name
> > >> > so
> > >> > > > that
> > >> > > > > > it
> > >> > > > > > > > contains "topic=mytopic,clientid=myclientid". That
> seems to
> > >> be
> > >> > a
> > >> > > > much
> > >> > > > > > > > cleaner way than having all parts included in a single
> > >> string
> > >> > > > > separated
> > >> > > > > > > by
> > >> > > > > > > > '|'.
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > >
> > >> > > > > > > > Jun
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > >> > > > > > > > mistry.p.bhavesh@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > HI Otis,
> > >> > > > > > > > >
> > >> > > > > > > > > What is migration path ?  If topic with special chars
> > >> exists
> > >> > > > > already(
> > >> > > > > > > > > ".","-","|" etc)  in previous version of
> > >> producer/consumer of
> > >> > > > > Kafka,
> > >> > > > > > > what
> > >> > > > > > > > > happens after the upgrade new producer or consumer
> (kafka
> > >> > > > version)
> > >> > > > > ?
> > >> > > > > > > > Also,
> > >> > > > > > > > > in new producer API (Kafka Trunk), does this enforce
> the
> > >> rule
> > >> > > > about
> > >> > > > > > > > client
> > >> > > > > > > > > id as well ?
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > >
> > >> > > > > > > > > Bhavesh
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > >> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi,
> > >> > > > > > > > > >
> > >> > > > > > > > > > So maybe I should I should have asked the Q
> explicitly:
> > >> > > > > > > > > > Could we commit the patch from
> > >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481
> now
> > >> > that, I
> > >> > > > > hope,
> > >> > > > > > > > it's
> > >> > > > > > > > > > clear what problems the current MBean names can
> cause?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Otis
> > >> > > > > > > > > > --
> > >> > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> Centralized
> > >> Log
> > >> > > > > > > Management
> > >> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > >> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Problem:*
> > >> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of
> things
> > >> > like
> > >> > > > > > > <consumer
> > >> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are
> > >> used
> > >> > as
> > >> > > > > > > > delimiters.
> > >> > > > > > > > > > >  When <consumer group> and <topic> don't contain
> the
> > >> > > > delimiter
> > >> > > > > > > > > character
> > >> > > > > > > > > > > all is good if you want to extract parts of this
> MBean
> > >> > name
> > >> > > > by
> > >> > > > > > > simply
> > >> > > > > > > > > > > splitting on the delimiter character.  The
> problem is
> > >> > that
> > >> > > > > dashes
> > >> > > > > > > are
> > >> > > > > > > > > > > allowed in topic and group names, so this
> splitting
> > >> > doesn't
> > >> > > > > work.
> > >> > > > > > > > > > > Moreover, underscores are also used as
> delimiters, and
> > >> > they
> > >> > > > can
> > >> > > > > > > also
> > >> > > > > > > > be
> > >> > > > > > > > > > > used in things like topic names.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Example*:
> > >> > > > > > > > > > > This MBean's name is composed of <consumer
> > >> > > > > > > > group>-<topic>-BytesPerSec:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > >> > > > > > > > > name="*myGroup**-myTopic**-*
> > >> > > > > > > > > > > BytesPerSec"
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Here we can actually split on "-" and extract all
> 3
> > >> parts
> > >> > > > from
> > >> > > > > > the
> > >> > > > > > > > > MBean
> > >> > > > > > > > > > > name::
> > >> > > > > > > > > > > * consumer group ('*myGroup*')
> > >> > > > > > > > > > > * topic ('*myTopic*')
> > >> > > > > > > > > > > * metric (‘BytesPerSec’)
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > All good!
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > But imagine if I named the group: *my-Group*
> > >> > > > > > > > > > > And if I named the topic: *my-Topic*
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Then we'd have:
> > >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > >> > > > > > > > > > name="*my-Group**-my-Topic**-*
> > >> > > > > > > > > > > BytesPerSec"
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Now splitting on "-" would no longer work!  To
> extract
> > >> > > > > "my-Group"
> > >> > > > > > > and
> > >> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to
> > >> know
> > >> > the
> > >> > > > > > > specific
> > >> > > > > > > > > > group
> > >> > > > > > > > > > > name and topic name to look for and could not use
> > >> generic
> > >> > > > > > approach
> > >> > > > > > > of
> > >> > > > > > > > > > just
> > >> > > > > > > > > > > splitting the MBean name on the delimiter.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > *Solution*:
> > >> > > > > > > > > > > The patch in
> > >> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> > >> > > > > > > > replaces
> > >> > > > > > > > > > > all _ and - characters where they are used as
> > >> delimiters
> > >> > in
> > >> > > > > MBean
> > >> > > > > > > > names
> > >> > > > > > > > > > > with a "|" character.  Because the "I" character
> is
> > >> not
> > >> > > > allowed
> > >> > > > > > in
> > >> > > > > > > > > topic
> > >> > > > > > > > > > > names, consumer groups, host names, splitting on
> this
> > >> new
> > >> > > and
> > >> > > > > > > unified
> > >> > > > > > > > > > > delimiter works.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I hope this explains the problem, the solution,
> and
> > >> that
> > >> > > this
> > >> > > > > can
> > >> > > > > > > > make
> > >> > > > > > > > > it
> > >> > > > > > > > > > > in the next 0.8.x.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Otis
> > >> > > > > > > > > > > --
> > >> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> > >> Centralized
> > >> > Log
> > >> > > > > > > > Management
> > >> > > > > > > > > > > Solr & Elasticsearch Support *
> http://sematext.com/
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
>
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Joel Koshy <jj...@gmail.com>.

That sounds good, although is that the only change (sorry I have not
done a careful review of that patch and would like to before it gets
checked in).

On Fri, Oct 31, 2014 at 10:42:13AM -0700, Jun Rao wrote:
> To circle back on this thread. The patch in kafka-1482 is almost ready. To
> make the mbean names more meaningful and easier to parse, the patch will
> use explicit key/value pairs in the mbean name for things like clientId and
> topic, and will get rid of the quotes.
> 
> So, instead of
>    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> we will have
>    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> 
> Any objection to committing this to the 0.8.2 branch?
> 
> Thanks,
> 
> Jun
> 
> On Fri, Oct 17, 2014 at 11:54 AM, Jun Rao <ju...@gmail.com> wrote:
> 
> > Hi, everyone,
> >
> > We are fixing the mbean names in kafka-1482, by adding separate explicit
> > tags in the name for things like clientId and topic. Another thing that
> > some people have complained before is that we use quotes in the jmx name.
> > Should we also just get rid of the quotes as part of kafka-1482? So,
> > instead of
> >    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > we will have
> >    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <ne...@gmail.com>
> > wrote:
> >
> >> I am going to vote for 1482 to be included in 0.8.2, if we have a patch
> >> submitted in a week. I think we've had this JIRA opened for too long and
> >> we
> >> held people back so it's only fair to release this.
> >>
> >> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
> >>
> >> > Otis,
> >> >
> >> > Just have the patch ready asap. We can make a call then.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> >> > otis.gospodnetic@gmail.com
> >> > > wrote:
> >> >
> >> > > Hi Jun,
> >> > >
> >> > > Would by the end of next week be acceptable for 0.8.2?
> >> > >
> >> > > Thanks,
> >> > > Otis
> >> > > --
> >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> > > Solr & Elasticsearch Support * http://sematext.com/
> >> > >
> >> > >
> >> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> >> > >
> >> > > > Otis,
> >> > > >
> >> > > > Yes, if you guys can help provide a patch in a few days, we can
> >> > probably
> >> > > > get it to the 0.8.2 release.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> >> > > > otis.gospodnetic@gmail.com> wrote:
> >> > > >
> >> > > > > Hi Jun,
> >> > > > >
> >> > > > > I think your MBean renaming approach will work.  I see
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version
> >> > > 0.8.2,
> >> > > > > but
> >> > > > > is not marked as a Blocker.  We'd love to get the MBeans fixed so
> >> > this
> >> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
> >> plate
> >> > > (the
> >> > > > > issue is currently Unassigned)?  If not, should we provide a new
> >> > patch
> >> > > > that
> >> > > > > uses your approach?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Otis
> >> > > > > --
> >> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> > Management
> >> > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
> >> wrote:
> >> > > > >
> >> > > > > > Otis,
> >> > > > > >
> >> > > > > > In kafka-1481, we will have to change the mbean names (at least
> >> the
> >> > > > ones
> >> > > > > > with clientid and topic) anyway. Using the name/value pair in
> >> the
> >> > > mbean
> >> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
> >> allowed
> >> > > in
> >> > > > > > clientid or topic.
> >> > > > > >
> >> > > > > > Bhavesh,
> >> > > > > >
> >> > > > > > Yes, I was thinking of making changes in the new metrics
> >> package.
> >> > > > > Something
> >> > > > > > like allowing the sensor names to have name/value pairs. The jmx
> >> > > names
> >> > > > > will
> >> > > > > > just follow accordingly. This is probably cleaner than doing the
> >> > > > > escaping.
> >> > > > > > Also, the metric names are more intuitive (otherwise, you have
> >> to
> >> > > know
> >> > > > > > which part is the clientid and which part is the topic).
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > Jun
> >> > > > > >
> >> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> >> > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > >
> >> > > > > > > Hi Jun,
> >> > > > > > >
> >> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com>
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > Bhavesh,
> >> > > > > > > >
> >> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit
> >> harder
> >> > to
> >> > > > > define
> >> > > > > > > the
> >> > > > > > > > JMX bean names. I see a couple of solutions here.
> >> > > > > > > >
> >> > > > > > > > 1. Disable dot in clientId and topic names. The issue is
> >> that
> >> > dot
> >> > > > may
> >> > > > > > > > already be used in existing deployment.
> >> > > > > > > >
> >> > > > > > > > 2. We can represent the JMX bean name differently in the new
> >> > > > > producer.
> >> > > > > > > > Instead of
> >> > > > > > > >   kafka.producer.myclientid:type=mytopic
> >> > > > > > > > we could change it to
> >> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> >> > > > > > > >
> >> > > > > > > > I felt that option 2 is probably better since it doesn't
> >> affect
> >> > > > > > existing
> >> > > > > > > > users.
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > If it doesn't affect existing users, great.
> >> > > > > > >
> >> > > > > > > If you are saying that each "piece" of MBean name could be
> >> > > expressed
> >> > > > as
> >> > > > > > > name=value pair, with something like "," (forbidden in host
> >> > names,
> >> > > > > topic
> >> > > > > > > names, client IDs, etc. I assume?) then yes, I think this
> >> would
> >> > be
> >> > > > > easier
> >> > > > > > > to parse and it would be easier for people to understand what
> >> is
> >> > > > what.
> >> > > > > > >
> >> > > > > > > Otis
> >> > > > > > > --
> >> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> > > > Management
> >> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > Otis,
> >> > > > > > > >
> >> > > > > > > > We probably can also use option 2 to address KAFKA-1481. For
> >> > > > > > > topic/clientid
> >> > > > > > > > specific metrics, we could explicitly specify the metric
> >> name
> >> > so
> >> > > > that
> >> > > > > > it
> >> > > > > > > > contains "topic=mytopic,clientid=myclientid". That seems to
> >> be
> >> > a
> >> > > > much
> >> > > > > > > > cleaner way than having all parts included in a single
> >> string
> >> > > > > separated
> >> > > > > > > by
> >> > > > > > > > '|'.
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > >
> >> > > > > > > > Jun
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> >> > > > > > > > mistry.p.bhavesh@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > HI Otis,
> >> > > > > > > > >
> >> > > > > > > > > What is migration path ?  If topic with special chars
> >> exists
> >> > > > > already(
> >> > > > > > > > > ".","-","|" etc)  in previous version of
> >> producer/consumer of
> >> > > > > Kafka,
> >> > > > > > > what
> >> > > > > > > > > happens after the upgrade new producer or consumer (kafka
> >> > > > version)
> >> > > > > ?
> >> > > > > > > > Also,
> >> > > > > > > > > in new producer API (Kafka Trunk), does this enforce the
> >> rule
> >> > > > about
> >> > > > > > > > client
> >> > > > > > > > > id as well ?
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Bhavesh
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> >> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > > So maybe I should I should have asked the Q explicitly:
> >> > > > > > > > > > Could we commit the patch from
> >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now
> >> > that, I
> >> > > > > hope,
> >> > > > > > > > it's
> >> > > > > > > > > > clear what problems the current MBean names can cause?
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Otis
> >> > > > > > > > > > --
> >> > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> >> Log
> >> > > > > > > Management
> >> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> >> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi,
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Problem:*
> >> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of things
> >> > like
> >> > > > > > > <consumer
> >> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are
> >> used
> >> > as
> >> > > > > > > > delimiters.
> >> > > > > > > > > > >  When <consumer group> and <topic> don't contain the
> >> > > > delimiter
> >> > > > > > > > > character
> >> > > > > > > > > > > all is good if you want to extract parts of this MBean
> >> > name
> >> > > > by
> >> > > > > > > simply
> >> > > > > > > > > > > splitting on the delimiter character.  The problem is
> >> > that
> >> > > > > dashes
> >> > > > > > > are
> >> > > > > > > > > > > allowed in topic and group names, so this splitting
> >> > doesn't
> >> > > > > work.
> >> > > > > > > > > > > Moreover, underscores are also used as delimiters, and
> >> > they
> >> > > > can
> >> > > > > > > also
> >> > > > > > > > be
> >> > > > > > > > > > > used in things like topic names.
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Example*:
> >> > > > > > > > > > > This MBean's name is composed of <consumer
> >> > > > > > > > group>-<topic>-BytesPerSec:
> >> > > > > > > > > > >
> >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> >> > > > > > > > > name="*myGroup**-myTopic**-*
> >> > > > > > > > > > > BytesPerSec"
> >> > > > > > > > > > >
> >> > > > > > > > > > > Here we can actually split on "-" and extract all 3
> >> parts
> >> > > > from
> >> > > > > > the
> >> > > > > > > > > MBean
> >> > > > > > > > > > > name::
> >> > > > > > > > > > > * consumer group ('*myGroup*')
> >> > > > > > > > > > > * topic ('*myTopic*')
> >> > > > > > > > > > > * metric (‘BytesPerSec’)
> >> > > > > > > > > > >
> >> > > > > > > > > > > All good!
> >> > > > > > > > > > >
> >> > > > > > > > > > > But imagine if I named the group: *my-Group*
> >> > > > > > > > > > > And if I named the topic: *my-Topic*
> >> > > > > > > > > > >
> >> > > > > > > > > > > Then we'd have:
> >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> >> > > > > > > > > > name="*my-Group**-my-Topic**-*
> >> > > > > > > > > > > BytesPerSec"
> >> > > > > > > > > > >
> >> > > > > > > > > > > Now splitting on "-" would no longer work!  To extract
> >> > > > > "my-Group"
> >> > > > > > > and
> >> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to
> >> know
> >> > the
> >> > > > > > > specific
> >> > > > > > > > > > group
> >> > > > > > > > > > > name and topic name to look for and could not use
> >> generic
> >> > > > > > approach
> >> > > > > > > of
> >> > > > > > > > > > just
> >> > > > > > > > > > > splitting the MBean name on the delimiter.
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Solution*:
> >> > > > > > > > > > > The patch in
> >> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> >> > > > > > > > replaces
> >> > > > > > > > > > > all _ and - characters where they are used as
> >> delimiters
> >> > in
> >> > > > > MBean
> >> > > > > > > > names
> >> > > > > > > > > > > with a "|" character.  Because the "I" character is
> >> not
> >> > > > allowed
> >> > > > > > in
> >> > > > > > > > > topic
> >> > > > > > > > > > > names, consumer groups, host names, splitting on this
> >> new
> >> > > and
> >> > > > > > > unified
> >> > > > > > > > > > > delimiter works.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I hope this explains the problem, the solution, and
> >> that
> >> > > this
> >> > > > > can
> >> > > > > > > > make
> >> > > > > > > > > it
> >> > > > > > > > > > > in the next 0.8.x.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Otis
> >> > > > > > > > > > > --
> >> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> >> Centralized
> >> > Log
> >> > > > > > > > Management
> >> > > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Joel Koshy <jj...@gmail.com>.

That sounds good, although is that the only change (sorry I have not
done a careful review of that patch and would like to before it gets
checked in).

On Fri, Oct 31, 2014 at 10:42:13AM -0700, Jun Rao wrote:
> To circle back on this thread. The patch in kafka-1482 is almost ready. To
> make the mbean names more meaningful and easier to parse, the patch will
> use explicit key/value pairs in the mbean name for things like clientId and
> topic, and will get rid of the quotes.
> 
> So, instead of
>    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> we will have
>    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> 
> Any objection to committing this to the 0.8.2 branch?
> 
> Thanks,
> 
> Jun
> 
> On Fri, Oct 17, 2014 at 11:54 AM, Jun Rao <ju...@gmail.com> wrote:
> 
> > Hi, everyone,
> >
> > We are fixing the mbean names in kafka-1482, by adding separate explicit
> > tags in the name for things like clientId and topic. Another thing that
> > some people have complained before is that we use quotes in the jmx name.
> > Should we also just get rid of the quotes as part of kafka-1482? So,
> > instead of
> >    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> > we will have
> >    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <ne...@gmail.com>
> > wrote:
> >
> >> I am going to vote for 1482 to be included in 0.8.2, if we have a patch
> >> submitted in a week. I think we've had this JIRA opened for too long and
> >> we
> >> held people back so it's only fair to release this.
> >>
> >> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
> >>
> >> > Otis,
> >> >
> >> > Just have the patch ready asap. We can make a call then.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> >> > otis.gospodnetic@gmail.com
> >> > > wrote:
> >> >
> >> > > Hi Jun,
> >> > >
> >> > > Would by the end of next week be acceptable for 0.8.2?
> >> > >
> >> > > Thanks,
> >> > > Otis
> >> > > --
> >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> > > Solr & Elasticsearch Support * http://sematext.com/
> >> > >
> >> > >
> >> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> >> > >
> >> > > > Otis,
> >> > > >
> >> > > > Yes, if you guys can help provide a patch in a few days, we can
> >> > probably
> >> > > > get it to the 0.8.2 release.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> >> > > > otis.gospodnetic@gmail.com> wrote:
> >> > > >
> >> > > > > Hi Jun,
> >> > > > >
> >> > > > > I think your MBean renaming approach will work.  I see
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version
> >> > > 0.8.2,
> >> > > > > but
> >> > > > > is not marked as a Blocker.  We'd love to get the MBeans fixed so
> >> > this
> >> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
> >> plate
> >> > > (the
> >> > > > > issue is currently Unassigned)?  If not, should we provide a new
> >> > patch
> >> > > > that
> >> > > > > uses your approach?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Otis
> >> > > > > --
> >> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> > Management
> >> > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
> >> wrote:
> >> > > > >
> >> > > > > > Otis,
> >> > > > > >
> >> > > > > > In kafka-1481, we will have to change the mbean names (at least
> >> the
> >> > > > ones
> >> > > > > > with clientid and topic) anyway. Using the name/value pair in
> >> the
> >> > > mbean
> >> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
> >> allowed
> >> > > in
> >> > > > > > clientid or topic.
> >> > > > > >
> >> > > > > > Bhavesh,
> >> > > > > >
> >> > > > > > Yes, I was thinking of making changes in the new metrics
> >> package.
> >> > > > > Something
> >> > > > > > like allowing the sensor names to have name/value pairs. The jmx
> >> > > names
> >> > > > > will
> >> > > > > > just follow accordingly. This is probably cleaner than doing the
> >> > > > > escaping.
> >> > > > > > Also, the metric names are more intuitive (otherwise, you have
> >> to
> >> > > know
> >> > > > > > which part is the clientid and which part is the topic).
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > Jun
> >> > > > > >
> >> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> >> > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > >
> >> > > > > > > Hi Jun,
> >> > > > > > >
> >> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com>
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > Bhavesh,
> >> > > > > > > >
> >> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit
> >> harder
> >> > to
> >> > > > > define
> >> > > > > > > the
> >> > > > > > > > JMX bean names. I see a couple of solutions here.
> >> > > > > > > >
> >> > > > > > > > 1. Disable dot in clientId and topic names. The issue is
> >> that
> >> > dot
> >> > > > may
> >> > > > > > > > already be used in existing deployment.
> >> > > > > > > >
> >> > > > > > > > 2. We can represent the JMX bean name differently in the new
> >> > > > > producer.
> >> > > > > > > > Instead of
> >> > > > > > > >   kafka.producer.myclientid:type=mytopic
> >> > > > > > > > we could change it to
> >> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> >> > > > > > > >
> >> > > > > > > > I felt that option 2 is probably better since it doesn't
> >> affect
> >> > > > > > existing
> >> > > > > > > > users.
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > If it doesn't affect existing users, great.
> >> > > > > > >
> >> > > > > > > If you are saying that each "piece" of MBean name could be
> >> > > expressed
> >> > > > as
> >> > > > > > > name=value pair, with something like "," (forbidden in host
> >> > names,
> >> > > > > topic
> >> > > > > > > names, client IDs, etc. I assume?) then yes, I think this
> >> would
> >> > be
> >> > > > > easier
> >> > > > > > > to parse and it would be easier for people to understand what
> >> is
> >> > > > what.
> >> > > > > > >
> >> > > > > > > Otis
> >> > > > > > > --
> >> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> > > > Management
> >> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > Otis,
> >> > > > > > > >
> >> > > > > > > > We probably can also use option 2 to address KAFKA-1481. For
> >> > > > > > > topic/clientid
> >> > > > > > > > specific metrics, we could explicitly specify the metric
> >> name
> >> > so
> >> > > > that
> >> > > > > > it
> >> > > > > > > > contains "topic=mytopic,clientid=myclientid". That seems to
> >> be
> >> > a
> >> > > > much
> >> > > > > > > > cleaner way than having all parts included in a single
> >> string
> >> > > > > separated
> >> > > > > > > by
> >> > > > > > > > '|'.
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > >
> >> > > > > > > > Jun
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> >> > > > > > > > mistry.p.bhavesh@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > HI Otis,
> >> > > > > > > > >
> >> > > > > > > > > What is migration path ?  If topic with special chars
> >> exists
> >> > > > > already(
> >> > > > > > > > > ".","-","|" etc)  in previous version of
> >> producer/consumer of
> >> > > > > Kafka,
> >> > > > > > > what
> >> > > > > > > > > happens after the upgrade new producer or consumer (kafka
> >> > > > version)
> >> > > > > ?
> >> > > > > > > > Also,
> >> > > > > > > > > in new producer API (Kafka Trunk), does this enforce the
> >> rule
> >> > > > about
> >> > > > > > > > client
> >> > > > > > > > > id as well ?
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Bhavesh
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> >> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > > So maybe I should I should have asked the Q explicitly:
> >> > > > > > > > > > Could we commit the patch from
> >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now
> >> > that, I
> >> > > > > hope,
> >> > > > > > > > it's
> >> > > > > > > > > > clear what problems the current MBean names can cause?
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Otis
> >> > > > > > > > > > --
> >> > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> >> Log
> >> > > > > > > Management
> >> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> >> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi,
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Problem:*
> >> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of things
> >> > like
> >> > > > > > > <consumer
> >> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are
> >> used
> >> > as
> >> > > > > > > > delimiters.
> >> > > > > > > > > > >  When <consumer group> and <topic> don't contain the
> >> > > > delimiter
> >> > > > > > > > > character
> >> > > > > > > > > > > all is good if you want to extract parts of this MBean
> >> > name
> >> > > > by
> >> > > > > > > simply
> >> > > > > > > > > > > splitting on the delimiter character.  The problem is
> >> > that
> >> > > > > dashes
> >> > > > > > > are
> >> > > > > > > > > > > allowed in topic and group names, so this splitting
> >> > doesn't
> >> > > > > work.
> >> > > > > > > > > > > Moreover, underscores are also used as delimiters, and
> >> > they
> >> > > > can
> >> > > > > > > also
> >> > > > > > > > be
> >> > > > > > > > > > > used in things like topic names.
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Example*:
> >> > > > > > > > > > > This MBean's name is composed of <consumer
> >> > > > > > > > group>-<topic>-BytesPerSec:
> >> > > > > > > > > > >
> >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> >> > > > > > > > > name="*myGroup**-myTopic**-*
> >> > > > > > > > > > > BytesPerSec"
> >> > > > > > > > > > >
> >> > > > > > > > > > > Here we can actually split on "-" and extract all 3
> >> parts
> >> > > > from
> >> > > > > > the
> >> > > > > > > > > MBean
> >> > > > > > > > > > > name::
> >> > > > > > > > > > > * consumer group ('*myGroup*')
> >> > > > > > > > > > > * topic ('*myTopic*')
> >> > > > > > > > > > > * metric (‘BytesPerSec’)
> >> > > > > > > > > > >
> >> > > > > > > > > > > All good!
> >> > > > > > > > > > >
> >> > > > > > > > > > > But imagine if I named the group: *my-Group*
> >> > > > > > > > > > > And if I named the topic: *my-Topic*
> >> > > > > > > > > > >
> >> > > > > > > > > > > Then we'd have:
> >> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> >> > > > > > > > > > name="*my-Group**-my-Topic**-*
> >> > > > > > > > > > > BytesPerSec"
> >> > > > > > > > > > >
> >> > > > > > > > > > > Now splitting on "-" would no longer work!  To extract
> >> > > > > "my-Group"
> >> > > > > > > and
> >> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to
> >> know
> >> > the
> >> > > > > > > specific
> >> > > > > > > > > > group
> >> > > > > > > > > > > name and topic name to look for and could not use
> >> generic
> >> > > > > > approach
> >> > > > > > > of
> >> > > > > > > > > > just
> >> > > > > > > > > > > splitting the MBean name on the delimiter.
> >> > > > > > > > > > >
> >> > > > > > > > > > > *Solution*:
> >> > > > > > > > > > > The patch in
> >> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> >> > > > > > > > replaces
> >> > > > > > > > > > > all _ and - characters where they are used as
> >> delimiters
> >> > in
> >> > > > > MBean
> >> > > > > > > > names
> >> > > > > > > > > > > with a "|" character.  Because the "I" character is
> >> not
> >> > > > allowed
> >> > > > > > in
> >> > > > > > > > > topic
> >> > > > > > > > > > > names, consumer groups, host names, splitting on this
> >> new
> >> > > and
> >> > > > > > > unified
> >> > > > > > > > > > > delimiter works.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I hope this explains the problem, the solution, and
> >> that
> >> > > this
> >> > > > > can
> >> > > > > > > > make
> >> > > > > > > > > it
> >> > > > > > > > > > > in the next 0.8.x.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Otis
> >> > > > > > > > > > > --
> >> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
> >> Centralized
> >> > Log
> >> > > > > > > > Management
> >> > > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

To circle back on this thread. The patch in kafka-1482 is almost ready. To
make the mbean names more meaningful and easier to parse, the patch will
use explicit key/value pairs in the mbean name for things like clientId and
topic, and will get rid of the quotes.

So, instead of
   "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
we will have
   kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1

Any objection to committing this to the 0.8.2 branch?

Thanks,

Jun

On Fri, Oct 17, 2014 at 11:54 AM, Jun Rao <ju...@gmail.com> wrote:

> Hi, everyone,
>
> We are fixing the mbean names in kafka-1482, by adding separate explicit
> tags in the name for things like clientId and topic. Another thing that
> some people have complained before is that we use quotes in the jmx name.
> Should we also just get rid of the quotes as part of kafka-1482? So,
> instead of
>    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> we will have
>    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
>
> Thanks,
>
> Jun
>
>
> On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <ne...@gmail.com>
> wrote:
>
>> I am going to vote for 1482 to be included in 0.8.2, if we have a patch
>> submitted in a week. I think we've had this JIRA opened for too long and
>> we
>> held people back so it's only fair to release this.
>>
>> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
>>
>> > Otis,
>> >
>> > Just have the patch ready asap. We can make a call then.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
>> > otis.gospodnetic@gmail.com
>> > > wrote:
>> >
>> > > Hi Jun,
>> > >
>> > > Would by the end of next week be acceptable for 0.8.2?
>> > >
>> > > Thanks,
>> > > Otis
>> > > --
>> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> > > Solr & Elasticsearch Support * http://sematext.com/
>> > >
>> > >
>> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
>> > >
>> > > > Otis,
>> > > >
>> > > > Yes, if you guys can help provide a patch in a few days, we can
>> > probably
>> > > > get it to the 0.8.2 release.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jun
>> > > >
>> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
>> > > > otis.gospodnetic@gmail.com> wrote:
>> > > >
>> > > > > Hi Jun,
>> > > > >
>> > > > > I think your MBean renaming approach will work.  I see
>> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version
>> > > 0.8.2,
>> > > > > but
>> > > > > is not marked as a Blocker.  We'd love to get the MBeans fixed so
>> > this
>> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
>> plate
>> > > (the
>> > > > > issue is currently Unassigned)?  If not, should we provide a new
>> > patch
>> > > > that
>> > > > > uses your approach?
>> > > > >
>> > > > > Thanks,
>> > > > > Otis
>> > > > > --
>> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> > Management
>> > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > >
>> > > > >
>> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
>> wrote:
>> > > > >
>> > > > > > Otis,
>> > > > > >
>> > > > > > In kafka-1481, we will have to change the mbean names (at least
>> the
>> > > > ones
>> > > > > > with clientid and topic) anyway. Using the name/value pair in
>> the
>> > > mbean
>> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
>> allowed
>> > > in
>> > > > > > clientid or topic.
>> > > > > >
>> > > > > > Bhavesh,
>> > > > > >
>> > > > > > Yes, I was thinking of making changes in the new metrics
>> package.
>> > > > > Something
>> > > > > > like allowing the sensor names to have name/value pairs. The jmx
>> > > names
>> > > > > will
>> > > > > > just follow accordingly. This is probably cleaner than doing the
>> > > > > escaping.
>> > > > > > Also, the metric names are more intuitive (otherwise, you have
>> to
>> > > know
>> > > > > > which part is the clientid and which part is the topic).
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Jun
>> > > > > >
>> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
>> > > > > > otis.gospodnetic@gmail.com> wrote:
>> > > > > >
>> > > > > > > Hi Jun,
>> > > > > > >
>> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com>
>> > > wrote:
>> > > > > > >
>> > > > > > > > Bhavesh,
>> > > > > > > >
>> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit
>> harder
>> > to
>> > > > > define
>> > > > > > > the
>> > > > > > > > JMX bean names. I see a couple of solutions here.
>> > > > > > > >
>> > > > > > > > 1. Disable dot in clientId and topic names. The issue is
>> that
>> > dot
>> > > > may
>> > > > > > > > already be used in existing deployment.
>> > > > > > > >
>> > > > > > > > 2. We can represent the JMX bean name differently in the new
>> > > > > producer.
>> > > > > > > > Instead of
>> > > > > > > >   kafka.producer.myclientid:type=mytopic
>> > > > > > > > we could change it to
>> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
>> > > > > > > >
>> > > > > > > > I felt that option 2 is probably better since it doesn't
>> affect
>> > > > > > existing
>> > > > > > > > users.
>> > > > > > > >
>> > > > > > >
>> > > > > > > If it doesn't affect existing users, great.
>> > > > > > >
>> > > > > > > If you are saying that each "piece" of MBean name could be
>> > > expressed
>> > > > as
>> > > > > > > name=value pair, with something like "," (forbidden in host
>> > names,
>> > > > > topic
>> > > > > > > names, client IDs, etc. I assume?) then yes, I think this
>> would
>> > be
>> > > > > easier
>> > > > > > > to parse and it would be easier for people to understand what
>> is
>> > > > what.
>> > > > > > >
>> > > > > > > Otis
>> > > > > > > --
>> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> > > > Management
>> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > >
>> > > > > > > > Otis,
>> > > > > > > >
>> > > > > > > > We probably can also use option 2 to address KAFKA-1481. For
>> > > > > > > topic/clientid
>> > > > > > > > specific metrics, we could explicitly specify the metric
>> name
>> > so
>> > > > that
>> > > > > > it
>> > > > > > > > contains "topic=mytopic,clientid=myclientid". That seems to
>> be
>> > a
>> > > > much
>> > > > > > > > cleaner way than having all parts included in a single
>> string
>> > > > > separated
>> > > > > > > by
>> > > > > > > > '|'.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > >
>> > > > > > > > Jun
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
>> > > > > > > > mistry.p.bhavesh@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > HI Otis,
>> > > > > > > > >
>> > > > > > > > > What is migration path ?  If topic with special chars
>> exists
>> > > > > already(
>> > > > > > > > > ".","-","|" etc)  in previous version of
>> producer/consumer of
>> > > > > Kafka,
>> > > > > > > what
>> > > > > > > > > happens after the upgrade new producer or consumer (kafka
>> > > > version)
>> > > > > ?
>> > > > > > > > Also,
>> > > > > > > > > in new producer API (Kafka Trunk), does this enforce the
>> rule
>> > > > about
>> > > > > > > > client
>> > > > > > > > > id as well ?
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Bhavesh
>> > > > > > > > >
>> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
>> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi,
>> > > > > > > > > >
>> > > > > > > > > > So maybe I should I should have asked the Q explicitly:
>> > > > > > > > > > Could we commit the patch from
>> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now
>> > that, I
>> > > > > hope,
>> > > > > > > > it's
>> > > > > > > > > > clear what problems the current MBean names can cause?
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Otis
>> > > > > > > > > > --
>> > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
>> Log
>> > > > > > > Management
>> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
>> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi,
>> > > > > > > > > > >
>> > > > > > > > > > > *Problem:*
>> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of things
>> > like
>> > > > > > > <consumer
>> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are
>> used
>> > as
>> > > > > > > > delimiters.
>> > > > > > > > > > >  When <consumer group> and <topic> don't contain the
>> > > > delimiter
>> > > > > > > > > character
>> > > > > > > > > > > all is good if you want to extract parts of this MBean
>> > name
>> > > > by
>> > > > > > > simply
>> > > > > > > > > > > splitting on the delimiter character.  The problem is
>> > that
>> > > > > dashes
>> > > > > > > are
>> > > > > > > > > > > allowed in topic and group names, so this splitting
>> > doesn't
>> > > > > work.
>> > > > > > > > > > > Moreover, underscores are also used as delimiters, and
>> > they
>> > > > can
>> > > > > > > also
>> > > > > > > > be
>> > > > > > > > > > > used in things like topic names.
>> > > > > > > > > > >
>> > > > > > > > > > > *Example*:
>> > > > > > > > > > > This MBean's name is composed of <consumer
>> > > > > > > > group>-<topic>-BytesPerSec:
>> > > > > > > > > > >
>> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
>> > > > > > > > > name="*myGroup**-myTopic**-*
>> > > > > > > > > > > BytesPerSec"
>> > > > > > > > > > >
>> > > > > > > > > > > Here we can actually split on "-" and extract all 3
>> parts
>> > > > from
>> > > > > > the
>> > > > > > > > > MBean
>> > > > > > > > > > > name::
>> > > > > > > > > > > * consumer group ('*myGroup*')
>> > > > > > > > > > > * topic ('*myTopic*')
>> > > > > > > > > > > * metric (‘BytesPerSec’)
>> > > > > > > > > > >
>> > > > > > > > > > > All good!
>> > > > > > > > > > >
>> > > > > > > > > > > But imagine if I named the group: *my-Group*
>> > > > > > > > > > > And if I named the topic: *my-Topic*
>> > > > > > > > > > >
>> > > > > > > > > > > Then we'd have:
>> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
>> > > > > > > > > > name="*my-Group**-my-Topic**-*
>> > > > > > > > > > > BytesPerSec"
>> > > > > > > > > > >
>> > > > > > > > > > > Now splitting on "-" would no longer work!  To extract
>> > > > > "my-Group"
>> > > > > > > and
>> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to
>> know
>> > the
>> > > > > > > specific
>> > > > > > > > > > group
>> > > > > > > > > > > name and topic name to look for and could not use
>> generic
>> > > > > > approach
>> > > > > > > of
>> > > > > > > > > > just
>> > > > > > > > > > > splitting the MBean name on the delimiter.
>> > > > > > > > > > >
>> > > > > > > > > > > *Solution*:
>> > > > > > > > > > > The patch in
>> > > > https://issues.apache.org/jira/browse/KAFKA-1481
>> > > > > > > > replaces
>> > > > > > > > > > > all _ and - characters where they are used as
>> delimiters
>> > in
>> > > > > MBean
>> > > > > > > > names
>> > > > > > > > > > > with a "|" character.  Because the "I" character is
>> not
>> > > > allowed
>> > > > > > in
>> > > > > > > > > topic
>> > > > > > > > > > > names, consumer groups, host names, splitting on this
>> new
>> > > and
>> > > > > > > unified
>> > > > > > > > > > > delimiter works.
>> > > > > > > > > > >
>> > > > > > > > > > > I hope this explains the problem, the solution, and
>> that
>> > > this
>> > > > > can
>> > > > > > > > make
>> > > > > > > > > it
>> > > > > > > > > > > in the next 0.8.x.
>> > > > > > > > > > >
>> > > > > > > > > > > Otis
>> > > > > > > > > > > --
>> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
>> Centralized
>> > Log
>> > > > > > > > Management
>> > > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Erik,

It seems that we can customized the mbean names with Metrics 2.2.0. Any
other reasons that we need to downgrade to Metrics 2.1.5?

Thanks,

Jun

On Sun, Nov 2, 2014 at 12:10 PM, Erik van Oosten <
e.vanoosten@grons.nl.invalid> wrote:

> Hi Jun,
>
> The quotes are because of a regression in Metrics 2.2.0. IMHO Metrics
> 2.2.0 should not be used because of this. Just downgrade to Metrics 2.1.5
> and you are good.
>
> Of course, upgrading to Metrics 3 would do the trick also.
>
> Kind regards,
>     Erik.
>
>
> Jun Rao schreef op 17-10-14 om 20:54:
>
>  Hi, everyone,
>>
>> We are fixing the mbean names in kafka-1482, by adding separate explicit
>> tags in the name for things like clientId and topic. Another thing that
>> some people have complained before is that we use quotes in the jmx name.
>> Should we also just get rid of the quotes as part of kafka-1482? So,
>> instead of
>>     "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
>> we will have
>>     kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
>>
>> Thanks,
>>
>> Jun
>>
>>
>
> --
> Erik van Oosten
> http://www.day-to-day-stuff.blogspot.com/
>
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Erik van Oosten <e....@grons.nl.INVALID>.

Hi Jun,

The quotes are because of a regression in Metrics 2.2.0. IMHO Metrics 
2.2.0 should not be used because of this. Just downgrade to Metrics 
2.1.5 and you are good.

Of course, upgrading to Metrics 3 would do the trick also.

Kind regards,
     Erik.


Jun Rao schreef op 17-10-14 om 20:54:
> Hi, everyone,
>
> We are fixing the mbean names in kafka-1482, by adding separate explicit
> tags in the name for things like clientId and topic. Another thing that
> some people have complained before is that we use quotes in the jmx name.
> Should we also just get rid of the quotes as part of kafka-1482? So,
> instead of
>     "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> we will have
>     kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
>
> Thanks,
>
> Jun
>


-- 
Erik van Oosten
http://www.day-to-day-stuff.blogspot.com/

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

To circle back on this thread. The patch in kafka-1482 is almost ready. To
make the mbean names more meaningful and easier to parse, the patch will
use explicit key/value pairs in the mbean name for things like clientId and
topic, and will get rid of the quotes.

So, instead of
   "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
we will have
   kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1

Any objection to committing this to the 0.8.2 branch?

Thanks,

Jun

On Fri, Oct 17, 2014 at 11:54 AM, Jun Rao <ju...@gmail.com> wrote:

> Hi, everyone,
>
> We are fixing the mbean names in kafka-1482, by adding separate explicit
> tags in the name for things like clientId and topic. Another thing that
> some people have complained before is that we use quotes in the jmx name.
> Should we also just get rid of the quotes as part of kafka-1482? So,
> instead of
>    "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
> we will have
>    kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
>
> Thanks,
>
> Jun
>
>
> On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <ne...@gmail.com>
> wrote:
>
>> I am going to vote for 1482 to be included in 0.8.2, if we have a patch
>> submitted in a week. I think we've had this JIRA opened for too long and
>> we
>> held people back so it's only fair to release this.
>>
>> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
>>
>> > Otis,
>> >
>> > Just have the patch ready asap. We can make a call then.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
>> > otis.gospodnetic@gmail.com
>> > > wrote:
>> >
>> > > Hi Jun,
>> > >
>> > > Would by the end of next week be acceptable for 0.8.2?
>> > >
>> > > Thanks,
>> > > Otis
>> > > --
>> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> > > Solr & Elasticsearch Support * http://sematext.com/
>> > >
>> > >
>> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
>> > >
>> > > > Otis,
>> > > >
>> > > > Yes, if you guys can help provide a patch in a few days, we can
>> > probably
>> > > > get it to the 0.8.2 release.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jun
>> > > >
>> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
>> > > > otis.gospodnetic@gmail.com> wrote:
>> > > >
>> > > > > Hi Jun,
>> > > > >
>> > > > > I think your MBean renaming approach will work.  I see
>> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version
>> > > 0.8.2,
>> > > > > but
>> > > > > is not marked as a Blocker.  We'd love to get the MBeans fixed so
>> > this
>> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
>> plate
>> > > (the
>> > > > > issue is currently Unassigned)?  If not, should we provide a new
>> > patch
>> > > > that
>> > > > > uses your approach?
>> > > > >
>> > > > > Thanks,
>> > > > > Otis
>> > > > > --
>> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> > Management
>> > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > >
>> > > > >
>> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com>
>> wrote:
>> > > > >
>> > > > > > Otis,
>> > > > > >
>> > > > > > In kafka-1481, we will have to change the mbean names (at least
>> the
>> > > > ones
>> > > > > > with clientid and topic) anyway. Using the name/value pair in
>> the
>> > > mbean
>> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
>> allowed
>> > > in
>> > > > > > clientid or topic.
>> > > > > >
>> > > > > > Bhavesh,
>> > > > > >
>> > > > > > Yes, I was thinking of making changes in the new metrics
>> package.
>> > > > > Something
>> > > > > > like allowing the sensor names to have name/value pairs. The jmx
>> > > names
>> > > > > will
>> > > > > > just follow accordingly. This is probably cleaner than doing the
>> > > > > escaping.
>> > > > > > Also, the metric names are more intuitive (otherwise, you have
>> to
>> > > know
>> > > > > > which part is the clientid and which part is the topic).
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Jun
>> > > > > >
>> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
>> > > > > > otis.gospodnetic@gmail.com> wrote:
>> > > > > >
>> > > > > > > Hi Jun,
>> > > > > > >
>> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com>
>> > > wrote:
>> > > > > > >
>> > > > > > > > Bhavesh,
>> > > > > > > >
>> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit
>> harder
>> > to
>> > > > > define
>> > > > > > > the
>> > > > > > > > JMX bean names. I see a couple of solutions here.
>> > > > > > > >
>> > > > > > > > 1. Disable dot in clientId and topic names. The issue is
>> that
>> > dot
>> > > > may
>> > > > > > > > already be used in existing deployment.
>> > > > > > > >
>> > > > > > > > 2. We can represent the JMX bean name differently in the new
>> > > > > producer.
>> > > > > > > > Instead of
>> > > > > > > >   kafka.producer.myclientid:type=mytopic
>> > > > > > > > we could change it to
>> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
>> > > > > > > >
>> > > > > > > > I felt that option 2 is probably better since it doesn't
>> affect
>> > > > > > existing
>> > > > > > > > users.
>> > > > > > > >
>> > > > > > >
>> > > > > > > If it doesn't affect existing users, great.
>> > > > > > >
>> > > > > > > If you are saying that each "piece" of MBean name could be
>> > > expressed
>> > > > as
>> > > > > > > name=value pair, with something like "," (forbidden in host
>> > names,
>> > > > > topic
>> > > > > > > names, client IDs, etc. I assume?) then yes, I think this
>> would
>> > be
>> > > > > easier
>> > > > > > > to parse and it would be easier for people to understand what
>> is
>> > > > what.
>> > > > > > >
>> > > > > > > Otis
>> > > > > > > --
>> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> > > > Management
>> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > >
>> > > > > > > > Otis,
>> > > > > > > >
>> > > > > > > > We probably can also use option 2 to address KAFKA-1481. For
>> > > > > > > topic/clientid
>> > > > > > > > specific metrics, we could explicitly specify the metric
>> name
>> > so
>> > > > that
>> > > > > > it
>> > > > > > > > contains "topic=mytopic,clientid=myclientid". That seems to
>> be
>> > a
>> > > > much
>> > > > > > > > cleaner way than having all parts included in a single
>> string
>> > > > > separated
>> > > > > > > by
>> > > > > > > > '|'.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > >
>> > > > > > > > Jun
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
>> > > > > > > > mistry.p.bhavesh@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > HI Otis,
>> > > > > > > > >
>> > > > > > > > > What is migration path ?  If topic with special chars
>> exists
>> > > > > already(
>> > > > > > > > > ".","-","|" etc)  in previous version of
>> producer/consumer of
>> > > > > Kafka,
>> > > > > > > what
>> > > > > > > > > happens after the upgrade new producer or consumer (kafka
>> > > > version)
>> > > > > ?
>> > > > > > > > Also,
>> > > > > > > > > in new producer API (Kafka Trunk), does this enforce the
>> rule
>> > > > about
>> > > > > > > > client
>> > > > > > > > > id as well ?
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Bhavesh
>> > > > > > > > >
>> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
>> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi,
>> > > > > > > > > >
>> > > > > > > > > > So maybe I should I should have asked the Q explicitly:
>> > > > > > > > > > Could we commit the patch from
>> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now
>> > that, I
>> > > > > hope,
>> > > > > > > > it's
>> > > > > > > > > > clear what problems the current MBean names can cause?
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Otis
>> > > > > > > > > > --
>> > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
>> Log
>> > > > > > > Management
>> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
>> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi,
>> > > > > > > > > > >
>> > > > > > > > > > > *Problem:*
>> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of things
>> > like
>> > > > > > > <consumer
>> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are
>> used
>> > as
>> > > > > > > > delimiters.
>> > > > > > > > > > >  When <consumer group> and <topic> don't contain the
>> > > > delimiter
>> > > > > > > > > character
>> > > > > > > > > > > all is good if you want to extract parts of this MBean
>> > name
>> > > > by
>> > > > > > > simply
>> > > > > > > > > > > splitting on the delimiter character.  The problem is
>> > that
>> > > > > dashes
>> > > > > > > are
>> > > > > > > > > > > allowed in topic and group names, so this splitting
>> > doesn't
>> > > > > work.
>> > > > > > > > > > > Moreover, underscores are also used as delimiters, and
>> > they
>> > > > can
>> > > > > > > also
>> > > > > > > > be
>> > > > > > > > > > > used in things like topic names.
>> > > > > > > > > > >
>> > > > > > > > > > > *Example*:
>> > > > > > > > > > > This MBean's name is composed of <consumer
>> > > > > > > > group>-<topic>-BytesPerSec:
>> > > > > > > > > > >
>> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
>> > > > > > > > > name="*myGroup**-myTopic**-*
>> > > > > > > > > > > BytesPerSec"
>> > > > > > > > > > >
>> > > > > > > > > > > Here we can actually split on "-" and extract all 3
>> parts
>> > > > from
>> > > > > > the
>> > > > > > > > > MBean
>> > > > > > > > > > > name::
>> > > > > > > > > > > * consumer group ('*myGroup*')
>> > > > > > > > > > > * topic ('*myTopic*')
>> > > > > > > > > > > * metric (‘BytesPerSec’)
>> > > > > > > > > > >
>> > > > > > > > > > > All good!
>> > > > > > > > > > >
>> > > > > > > > > > > But imagine if I named the group: *my-Group*
>> > > > > > > > > > > And if I named the topic: *my-Topic*
>> > > > > > > > > > >
>> > > > > > > > > > > Then we'd have:
>> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
>> > > > > > > > > > name="*my-Group**-my-Topic**-*
>> > > > > > > > > > > BytesPerSec"
>> > > > > > > > > > >
>> > > > > > > > > > > Now splitting on "-" would no longer work!  To extract
>> > > > > "my-Group"
>> > > > > > > and
>> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to
>> know
>> > the
>> > > > > > > specific
>> > > > > > > > > > group
>> > > > > > > > > > > name and topic name to look for and could not use
>> generic
>> > > > > > approach
>> > > > > > > of
>> > > > > > > > > > just
>> > > > > > > > > > > splitting the MBean name on the delimiter.
>> > > > > > > > > > >
>> > > > > > > > > > > *Solution*:
>> > > > > > > > > > > The patch in
>> > > > https://issues.apache.org/jira/browse/KAFKA-1481
>> > > > > > > > replaces
>> > > > > > > > > > > all _ and - characters where they are used as
>> delimiters
>> > in
>> > > > > MBean
>> > > > > > > > names
>> > > > > > > > > > > with a "|" character.  Because the "I" character is
>> not
>> > > > allowed
>> > > > > > in
>> > > > > > > > > topic
>> > > > > > > > > > > names, consumer groups, host names, splitting on this
>> new
>> > > and
>> > > > > > > unified
>> > > > > > > > > > > delimiter works.
>> > > > > > > > > > >
>> > > > > > > > > > > I hope this explains the problem, the solution, and
>> that
>> > > this
>> > > > > can
>> > > > > > > > make
>> > > > > > > > > it
>> > > > > > > > > > > in the next 0.8.x.
>> > > > > > > > > > >
>> > > > > > > > > > > Otis
>> > > > > > > > > > > --
>> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection *
>> Centralized
>> > Log
>> > > > > > > > Management
>> > > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Hi, everyone,

We are fixing the mbean names in kafka-1482, by adding separate explicit
tags in the name for things like clientId and topic. Another thing that
some people have complained before is that we use quotes in the jmx name.
Should we also just get rid of the quotes as part of kafka-1482? So,
instead of
   "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec"
we will have
   kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1

Thanks,

Jun


On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <ne...@gmail.com>
wrote:

> I am going to vote for 1482 to be included in 0.8.2, if we have a patch
> submitted in a week. I think we've had this JIRA opened for too long and we
> held people back so it's only fair to release this.
>
> On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Otis,
> >
> > Just have the patch ready asap. We can make a call then.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com
> > > wrote:
> >
> > > Hi Jun,
> > >
> > > Would by the end of next week be acceptable for 0.8.2?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Otis,
> > > >
> > > > Yes, if you guys can help provide a patch in a few days, we can
> > probably
> > > > get it to the 0.8.2 release.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> > > > otis.gospodnetic@gmail.com> wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > I think your MBean renaming approach will work.  I see
> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version
> > > 0.8.2,
> > > > > but
> > > > > is not marked as a Blocker.  We'd love to get the MBeans fixed so
> > this
> > > > > makes it in 0.8.2 release.  Do you know if this is on anyone's
> plate
> > > (the
> > > > > issue is currently Unassigned)?  If not, should we provide a new
> > patch
> > > > that
> > > > > uses your approach?
> > > > >
> > > > > Thanks,
> > > > > Otis
> > > > > --
> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > >
> > > > >
> > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com> wrote:
> > > > >
> > > > > > Otis,
> > > > > >
> > > > > > In kafka-1481, we will have to change the mbean names (at least
> the
> > > > ones
> > > > > > with clientid and topic) anyway. Using the name/value pair in the
> > > mbean
> > > > > > name allows us to do this in a cleaner way. Yes, "," is not
> allowed
> > > in
> > > > > > clientid or topic.
> > > > > >
> > > > > > Bhavesh,
> > > > > >
> > > > > > Yes, I was thinking of making changes in the new metrics package.
> > > > > Something
> > > > > > like allowing the sensor names to have name/value pairs. The jmx
> > > names
> > > > > will
> > > > > > just follow accordingly. This is probably cleaner than doing the
> > > > > escaping.
> > > > > > Also, the metric names are more intuitive (otherwise, you have to
> > > know
> > > > > > which part is the clientid and which part is the topic).
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Bhavesh,
> > > > > > > >
> > > > > > > > Yes, allowing dot in clientId and topic makes it a bit harder
> > to
> > > > > define
> > > > > > > the
> > > > > > > > JMX bean names. I see a couple of solutions here.
> > > > > > > >
> > > > > > > > 1. Disable dot in clientId and topic names. The issue is that
> > dot
> > > > may
> > > > > > > > already be used in existing deployment.
> > > > > > > >
> > > > > > > > 2. We can represent the JMX bean name differently in the new
> > > > > producer.
> > > > > > > > Instead of
> > > > > > > >   kafka.producer.myclientid:type=mytopic
> > > > > > > > we could change it to
> > > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > > > > > > >
> > > > > > > > I felt that option 2 is probably better since it doesn't
> affect
> > > > > > existing
> > > > > > > > users.
> > > > > > > >
> > > > > > >
> > > > > > > If it doesn't affect existing users, great.
> > > > > > >
> > > > > > > If you are saying that each "piece" of MBean name could be
> > > expressed
> > > > as
> > > > > > > name=value pair, with something like "," (forbidden in host
> > names,
> > > > > topic
> > > > > > > names, client IDs, etc. I assume?) then yes, I think this would
> > be
> > > > > easier
> > > > > > > to parse and it would be easier for people to understand what
> is
> > > > what.
> > > > > > >
> > > > > > > Otis
> > > > > > > --
> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > > Management
> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Otis,
> > > > > > > >
> > > > > > > > We probably can also use option 2 to address KAFKA-1481. For
> > > > > > > topic/clientid
> > > > > > > > specific metrics, we could explicitly specify the metric name
> > so
> > > > that
> > > > > > it
> > > > > > > > contains "topic=mytopic,clientid=myclientid". That seems to
> be
> > a
> > > > much
> > > > > > > > cleaner way than having all parts included in a single string
> > > > > separated
> > > > > > > by
> > > > > > > > '|'.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > > > > > > > mistry.p.bhavesh@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > HI Otis,
> > > > > > > > >
> > > > > > > > > What is migration path ?  If topic with special chars
> exists
> > > > > already(
> > > > > > > > > ".","-","|" etc)  in previous version of producer/consumer
> of
> > > > > Kafka,
> > > > > > > what
> > > > > > > > > happens after the upgrade new producer or consumer (kafka
> > > > version)
> > > > > ?
> > > > > > > > Also,
> > > > > > > > > in new producer API (Kafka Trunk), does this enforce the
> rule
> > > > about
> > > > > > > > client
> > > > > > > > > id as well ?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Bhavesh
> > > > > > > > >
> > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > So maybe I should I should have asked the Q explicitly:
> > > > > > > > > > Could we commit the patch from
> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now
> > that, I
> > > > > hope,
> > > > > > > > it's
> > > > > > > > > > clear what problems the current MBean names can cause?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Otis
> > > > > > > > > > --
> > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> Log
> > > > > > > Management
> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > *Problem:*
> > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of things
> > like
> > > > > > > <consumer
> > > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are used
> > as
> > > > > > > > delimiters.
> > > > > > > > > > >  When <consumer group> and <topic> don't contain the
> > > > delimiter
> > > > > > > > > character
> > > > > > > > > > > all is good if you want to extract parts of this MBean
> > name
> > > > by
> > > > > > > simply
> > > > > > > > > > > splitting on the delimiter character.  The problem is
> > that
> > > > > dashes
> > > > > > > are
> > > > > > > > > > > allowed in topic and group names, so this splitting
> > doesn't
> > > > > work.
> > > > > > > > > > > Moreover, underscores are also used as delimiters, and
> > they
> > > > can
> > > > > > > also
> > > > > > > > be
> > > > > > > > > > > used in things like topic names.
> > > > > > > > > > >
> > > > > > > > > > > *Example*:
> > > > > > > > > > > This MBean's name is composed of <consumer
> > > > > > > > group>-<topic>-BytesPerSec:
> > > > > > > > > > >
> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > > > name="*myGroup**-myTopic**-*
> > > > > > > > > > > BytesPerSec"
> > > > > > > > > > >
> > > > > > > > > > > Here we can actually split on "-" and extract all 3
> parts
> > > > from
> > > > > > the
> > > > > > > > > MBean
> > > > > > > > > > > name::
> > > > > > > > > > > * consumer group ('*myGroup*')
> > > > > > > > > > > * topic ('*myTopic*')
> > > > > > > > > > > * metric (‘BytesPerSec’)
> > > > > > > > > > >
> > > > > > > > > > > All good!
> > > > > > > > > > >
> > > > > > > > > > > But imagine if I named the group: *my-Group*
> > > > > > > > > > > And if I named the topic: *my-Topic*
> > > > > > > > > > >
> > > > > > > > > > > Then we'd have:
> > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > > > > name="*my-Group**-my-Topic**-*
> > > > > > > > > > > BytesPerSec"
> > > > > > > > > > >
> > > > > > > > > > > Now splitting on "-" would no longer work!  To extract
> > > > > "my-Group"
> > > > > > > and
> > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to know
> > the
> > > > > > > specific
> > > > > > > > > > group
> > > > > > > > > > > name and topic name to look for and could not use
> generic
> > > > > > approach
> > > > > > > of
> > > > > > > > > > just
> > > > > > > > > > > splitting the MBean name on the delimiter.
> > > > > > > > > > >
> > > > > > > > > > > *Solution*:
> > > > > > > > > > > The patch in
> > > > https://issues.apache.org/jira/browse/KAFKA-1481
> > > > > > > > replaces
> > > > > > > > > > > all _ and - characters where they are used as
> delimiters
> > in
> > > > > MBean
> > > > > > > > names
> > > > > > > > > > > with a "|" character.  Because the "I" character is not
> > > > allowed
> > > > > > in
> > > > > > > > > topic
> > > > > > > > > > > names, consumer groups, host names, splitting on this
> new
> > > and
> > > > > > > unified
> > > > > > > > > > > delimiter works.
> > > > > > > > > > >
> > > > > > > > > > > I hope this explains the problem, the solution, and
> that
> > > this
> > > > > can
> > > > > > > > make
> > > > > > > > > it
> > > > > > > > > > > in the next 0.8.x.
> > > > > > > > > > >
> > > > > > > > > > > Otis
> > > > > > > > > > > --
> > > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> > Log
> > > > > > > > Management
> > > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Neha Narkhede <ne...@gmail.com>.

I am going to vote for 1482 to be included in 0.8.2, if we have a patch
submitted in a week. I think we've had this JIRA opened for too long and we
held people back so it's only fair to release this.

On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <ju...@gmail.com> wrote:

> Otis,
>
> Just have the patch ready asap. We can make a call then.
>
> Thanks,
>
> Jun
>
> On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com
> > wrote:
>
> > Hi Jun,
> >
> > Would by the end of next week be acceptable for 0.8.2?
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Otis,
> > >
> > > Yes, if you guys can help provide a patch in a few days, we can
> probably
> > > get it to the 0.8.2 release.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > I think your MBean renaming approach will work.  I see
> > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version
> > 0.8.2,
> > > > but
> > > > is not marked as a Blocker.  We'd love to get the MBeans fixed so
> this
> > > > makes it in 0.8.2 release.  Do you know if this is on anyone's plate
> > (the
> > > > issue is currently Unassigned)?  If not, should we provide a new
> patch
> > > that
> > > > uses your approach?
> > > >
> > > > Thanks,
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > > >
> > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > > > Otis,
> > > > >
> > > > > In kafka-1481, we will have to change the mbean names (at least the
> > > ones
> > > > > with clientid and topic) anyway. Using the name/value pair in the
> > mbean
> > > > > name allows us to do this in a cleaner way. Yes, "," is not allowed
> > in
> > > > > clientid or topic.
> > > > >
> > > > > Bhavesh,
> > > > >
> > > > > Yes, I was thinking of making changes in the new metrics package.
> > > > Something
> > > > > like allowing the sensor names to have name/value pairs. The jmx
> > names
> > > > will
> > > > > just follow accordingly. This is probably cleaner than doing the
> > > > escaping.
> > > > > Also, the metric names are more intuitive (otherwise, you have to
> > know
> > > > > which part is the clientid and which part is the topic).
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > > > > otis.gospodnetic@gmail.com> wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Bhavesh,
> > > > > > >
> > > > > > > Yes, allowing dot in clientId and topic makes it a bit harder
> to
> > > > define
> > > > > > the
> > > > > > > JMX bean names. I see a couple of solutions here.
> > > > > > >
> > > > > > > 1. Disable dot in clientId and topic names. The issue is that
> dot
> > > may
> > > > > > > already be used in existing deployment.
> > > > > > >
> > > > > > > 2. We can represent the JMX bean name differently in the new
> > > > producer.
> > > > > > > Instead of
> > > > > > >   kafka.producer.myclientid:type=mytopic
> > > > > > > we could change it to
> > > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > > > > > >
> > > > > > > I felt that option 2 is probably better since it doesn't affect
> > > > > existing
> > > > > > > users.
> > > > > > >
> > > > > >
> > > > > > If it doesn't affect existing users, great.
> > > > > >
> > > > > > If you are saying that each "piece" of MBean name could be
> > expressed
> > > as
> > > > > > name=value pair, with something like "," (forbidden in host
> names,
> > > > topic
> > > > > > names, client IDs, etc. I assume?) then yes, I think this would
> be
> > > > easier
> > > > > > to parse and it would be easier for people to understand what is
> > > what.
> > > > > >
> > > > > > Otis
> > > > > > --
> > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Otis,
> > > > > > >
> > > > > > > We probably can also use option 2 to address KAFKA-1481. For
> > > > > > topic/clientid
> > > > > > > specific metrics, we could explicitly specify the metric name
> so
> > > that
> > > > > it
> > > > > > > contains "topic=mytopic,clientid=myclientid". That seems to be
> a
> > > much
> > > > > > > cleaner way than having all parts included in a single string
> > > > separated
> > > > > > by
> > > > > > > '|'.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > > > > > > mistry.p.bhavesh@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > HI Otis,
> > > > > > > >
> > > > > > > > What is migration path ?  If topic with special chars exists
> > > > already(
> > > > > > > > ".","-","|" etc)  in previous version of producer/consumer of
> > > > Kafka,
> > > > > > what
> > > > > > > > happens after the upgrade new producer or consumer (kafka
> > > version)
> > > > ?
> > > > > > > Also,
> > > > > > > > in new producer API (Kafka Trunk), does this enforce the rule
> > > about
> > > > > > > client
> > > > > > > > id as well ?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Bhavesh
> > > > > > > >
> > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > So maybe I should I should have asked the Q explicitly:
> > > > > > > > > Could we commit the patch from
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now
> that, I
> > > > hope,
> > > > > > > it's
> > > > > > > > > clear what problems the current MBean names can cause?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Otis
> > > > > > > > > --
> > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > > > > Management
> > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > *Problem:*
> > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of things
> like
> > > > > > <consumer
> > > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are used
> as
> > > > > > > delimiters.
> > > > > > > > > >  When <consumer group> and <topic> don't contain the
> > > delimiter
> > > > > > > > character
> > > > > > > > > > all is good if you want to extract parts of this MBean
> name
> > > by
> > > > > > simply
> > > > > > > > > > splitting on the delimiter character.  The problem is
> that
> > > > dashes
> > > > > > are
> > > > > > > > > > allowed in topic and group names, so this splitting
> doesn't
> > > > work.
> > > > > > > > > > Moreover, underscores are also used as delimiters, and
> they
> > > can
> > > > > > also
> > > > > > > be
> > > > > > > > > > used in things like topic names.
> > > > > > > > > >
> > > > > > > > > > *Example*:
> > > > > > > > > > This MBean's name is composed of <consumer
> > > > > > > group>-<topic>-BytesPerSec:
> > > > > > > > > >
> > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > > name="*myGroup**-myTopic**-*
> > > > > > > > > > BytesPerSec"
> > > > > > > > > >
> > > > > > > > > > Here we can actually split on "-" and extract all 3 parts
> > > from
> > > > > the
> > > > > > > > MBean
> > > > > > > > > > name::
> > > > > > > > > > * consumer group ('*myGroup*')
> > > > > > > > > > * topic ('*myTopic*')
> > > > > > > > > > * metric (‘BytesPerSec’)
> > > > > > > > > >
> > > > > > > > > > All good!
> > > > > > > > > >
> > > > > > > > > > But imagine if I named the group: *my-Group*
> > > > > > > > > > And if I named the topic: *my-Topic*
> > > > > > > > > >
> > > > > > > > > > Then we'd have:
> > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > > > name="*my-Group**-my-Topic**-*
> > > > > > > > > > BytesPerSec"
> > > > > > > > > >
> > > > > > > > > > Now splitting on "-" would no longer work!  To extract
> > > > "my-Group"
> > > > > > and
> > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to know
> the
> > > > > > specific
> > > > > > > > > group
> > > > > > > > > > name and topic name to look for and could not use generic
> > > > > approach
> > > > > > of
> > > > > > > > > just
> > > > > > > > > > splitting the MBean name on the delimiter.
> > > > > > > > > >
> > > > > > > > > > *Solution*:
> > > > > > > > > > The patch in
> > > https://issues.apache.org/jira/browse/KAFKA-1481
> > > > > > > replaces
> > > > > > > > > > all _ and - characters where they are used as delimiters
> in
> > > > MBean
> > > > > > > names
> > > > > > > > > > with a "|" character.  Because the "I" character is not
> > > allowed
> > > > > in
> > > > > > > > topic
> > > > > > > > > > names, consumer groups, host names, splitting on this new
> > and
> > > > > > unified
> > > > > > > > > > delimiter works.
> > > > > > > > > >
> > > > > > > > > > I hope this explains the problem, the solution, and that
> > this
> > > > can
> > > > > > > make
> > > > > > > > it
> > > > > > > > > > in the next 0.8.x.
> > > > > > > > > >
> > > > > > > > > > Otis
> > > > > > > > > > --
> > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized
> Log
> > > > > > > Management
> > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Otis,

Just have the patch ready asap. We can make a call then.

Thanks,

Jun

On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic <otis.gospodnetic@gmail.com
> wrote:

> Hi Jun,
>
> Would by the end of next week be acceptable for 0.8.2?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Otis,
> >
> > Yes, if you guys can help provide a patch in a few days, we can probably
> > get it to the 0.8.2 release.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi Jun,
> > >
> > > I think your MBean renaming approach will work.  I see
> > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version
> 0.8.2,
> > > but
> > > is not marked as a Blocker.  We'd love to get the MBeans fixed so this
> > > makes it in 0.8.2 release.  Do you know if this is on anyone's plate
> (the
> > > issue is currently Unassigned)?  If not, should we provide a new patch
> > that
> > > uses your approach?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Otis,
> > > >
> > > > In kafka-1481, we will have to change the mbean names (at least the
> > ones
> > > > with clientid and topic) anyway. Using the name/value pair in the
> mbean
> > > > name allows us to do this in a cleaner way. Yes, "," is not allowed
> in
> > > > clientid or topic.
> > > >
> > > > Bhavesh,
> > > >
> > > > Yes, I was thinking of making changes in the new metrics package.
> > > Something
> > > > like allowing the sensor names to have name/value pairs. The jmx
> names
> > > will
> > > > just follow accordingly. This is probably cleaner than doing the
> > > escaping.
> > > > Also, the metric names are more intuitive (otherwise, you have to
> know
> > > > which part is the clientid and which part is the topic).
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > > > otis.gospodnetic@gmail.com> wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com>
> wrote:
> > > > >
> > > > > > Bhavesh,
> > > > > >
> > > > > > Yes, allowing dot in clientId and topic makes it a bit harder to
> > > define
> > > > > the
> > > > > > JMX bean names. I see a couple of solutions here.
> > > > > >
> > > > > > 1. Disable dot in clientId and topic names. The issue is that dot
> > may
> > > > > > already be used in existing deployment.
> > > > > >
> > > > > > 2. We can represent the JMX bean name differently in the new
> > > producer.
> > > > > > Instead of
> > > > > >   kafka.producer.myclientid:type=mytopic
> > > > > > we could change it to
> > > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > > > > >
> > > > > > I felt that option 2 is probably better since it doesn't affect
> > > > existing
> > > > > > users.
> > > > > >
> > > > >
> > > > > If it doesn't affect existing users, great.
> > > > >
> > > > > If you are saying that each "piece" of MBean name could be
> expressed
> > as
> > > > > name=value pair, with something like "," (forbidden in host names,
> > > topic
> > > > > names, client IDs, etc. I assume?) then yes, I think this would be
> > > easier
> > > > > to parse and it would be easier for people to understand what is
> > what.
> > > > >
> > > > > Otis
> > > > > --
> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > Otis,
> > > > > >
> > > > > > We probably can also use option 2 to address KAFKA-1481. For
> > > > > topic/clientid
> > > > > > specific metrics, we could explicitly specify the metric name so
> > that
> > > > it
> > > > > > contains "topic=mytopic,clientid=myclientid". That seems to be a
> > much
> > > > > > cleaner way than having all parts included in a single string
> > > separated
> > > > > by
> > > > > > '|'.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > > > > > mistry.p.bhavesh@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > HI Otis,
> > > > > > >
> > > > > > > What is migration path ?  If topic with special chars exists
> > > already(
> > > > > > > ".","-","|" etc)  in previous version of producer/consumer of
> > > Kafka,
> > > > > what
> > > > > > > happens after the upgrade new producer or consumer (kafka
> > version)
> > > ?
> > > > > > Also,
> > > > > > > in new producer API (Kafka Trunk), does this enforce the rule
> > about
> > > > > > client
> > > > > > > id as well ?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Bhavesh
> > > > > > >
> > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > So maybe I should I should have asked the Q explicitly:
> > > > > > > > Could we commit the patch from
> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I
> > > hope,
> > > > > > it's
> > > > > > > > clear what problems the current MBean names can cause?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Otis
> > > > > > > > --
> > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > > > Management
> > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > *Problem:*
> > > > > > > > > Some Kafka 0.8.x MBeans have names composed of things like
> > > > > <consumer
> > > > > > > > > group>-<topic>-<metric name>.  Note how dashes are used as
> > > > > > delimiters.
> > > > > > > > >  When <consumer group> and <topic> don't contain the
> > delimiter
> > > > > > > character
> > > > > > > > > all is good if you want to extract parts of this MBean name
> > by
> > > > > simply
> > > > > > > > > splitting on the delimiter character.  The problem is that
> > > dashes
> > > > > are
> > > > > > > > > allowed in topic and group names, so this splitting doesn't
> > > work.
> > > > > > > > > Moreover, underscores are also used as delimiters, and they
> > can
> > > > > also
> > > > > > be
> > > > > > > > > used in things like topic names.
> > > > > > > > >
> > > > > > > > > *Example*:
> > > > > > > > > This MBean's name is composed of <consumer
> > > > > > group>-<topic>-BytesPerSec:
> > > > > > > > >
> > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > name="*myGroup**-myTopic**-*
> > > > > > > > > BytesPerSec"
> > > > > > > > >
> > > > > > > > > Here we can actually split on "-" and extract all 3 parts
> > from
> > > > the
> > > > > > > MBean
> > > > > > > > > name::
> > > > > > > > > * consumer group ('*myGroup*')
> > > > > > > > > * topic ('*myTopic*')
> > > > > > > > > * metric (‘BytesPerSec’)
> > > > > > > > >
> > > > > > > > > All good!
> > > > > > > > >
> > > > > > > > > But imagine if I named the group: *my-Group*
> > > > > > > > > And if I named the topic: *my-Topic*
> > > > > > > > >
> > > > > > > > > Then we'd have:
> > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > > name="*my-Group**-my-Topic**-*
> > > > > > > > > BytesPerSec"
> > > > > > > > >
> > > > > > > > > Now splitting on "-" would no longer work!  To extract
> > > "my-Group"
> > > > > and
> > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to know the
> > > > > specific
> > > > > > > > group
> > > > > > > > > name and topic name to look for and could not use generic
> > > > approach
> > > > > of
> > > > > > > > just
> > > > > > > > > splitting the MBean name on the delimiter.
> > > > > > > > >
> > > > > > > > > *Solution*:
> > > > > > > > > The patch in
> > https://issues.apache.org/jira/browse/KAFKA-1481
> > > > > > replaces
> > > > > > > > > all _ and - characters where they are used as delimiters in
> > > MBean
> > > > > > names
> > > > > > > > > with a "|" character.  Because the "I" character is not
> > allowed
> > > > in
> > > > > > > topic
> > > > > > > > > names, consumer groups, host names, splitting on this new
> and
> > > > > unified
> > > > > > > > > delimiter works.
> > > > > > > > >
> > > > > > > > > I hope this explains the problem, the solution, and that
> this
> > > can
> > > > > > make
> > > > > > > it
> > > > > > > > > in the next 0.8.x.
> > > > > > > > >
> > > > > > > > > Otis
> > > > > > > > > --
> > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > > > > Management
> > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Otis Gospodnetic <ot...@gmail.com>.

Hi Jun,

Would by the end of next week be acceptable for 0.8.2?

Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:

> Otis,
>
> Yes, if you guys can help provide a patch in a few days, we can probably
> get it to the 0.8.2 release.
>
> Thanks,
>
> Jun
>
> On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi Jun,
> >
> > I think your MBean renaming approach will work.  I see
> > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version 0.8.2,
> > but
> > is not marked as a Blocker.  We'd love to get the MBeans fixed so this
> > makes it in 0.8.2 release.  Do you know if this is on anyone's plate (the
> > issue is currently Unassigned)?  If not, should we provide a new patch
> that
> > uses your approach?
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Otis,
> > >
> > > In kafka-1481, we will have to change the mbean names (at least the
> ones
> > > with clientid and topic) anyway. Using the name/value pair in the mbean
> > > name allows us to do this in a cleaner way. Yes, "," is not allowed in
> > > clientid or topic.
> > >
> > > Bhavesh,
> > >
> > > Yes, I was thinking of making changes in the new metrics package.
> > Something
> > > like allowing the sensor names to have name/value pairs. The jmx names
> > will
> > > just follow accordingly. This is probably cleaner than doing the
> > escaping.
> > > Also, the metric names are more intuitive (otherwise, you have to know
> > > which part is the clientid and which part is the topic).
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > > > Bhavesh,
> > > > >
> > > > > Yes, allowing dot in clientId and topic makes it a bit harder to
> > define
> > > > the
> > > > > JMX bean names. I see a couple of solutions here.
> > > > >
> > > > > 1. Disable dot in clientId and topic names. The issue is that dot
> may
> > > > > already be used in existing deployment.
> > > > >
> > > > > 2. We can represent the JMX bean name differently in the new
> > producer.
> > > > > Instead of
> > > > >   kafka.producer.myclientid:type=mytopic
> > > > > we could change it to
> > > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > > > >
> > > > > I felt that option 2 is probably better since it doesn't affect
> > > existing
> > > > > users.
> > > > >
> > > >
> > > > If it doesn't affect existing users, great.
> > > >
> > > > If you are saying that each "piece" of MBean name could be expressed
> as
> > > > name=value pair, with something like "," (forbidden in host names,
> > topic
> > > > names, client IDs, etc. I assume?) then yes, I think this would be
> > easier
> > > > to parse and it would be easier for people to understand what is
> what.
> > > >
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > > >
> > > >
> > > > >
> > > > > Otis,
> > > > >
> > > > > We probably can also use option 2 to address KAFKA-1481. For
> > > > topic/clientid
> > > > > specific metrics, we could explicitly specify the metric name so
> that
> > > it
> > > > > contains "topic=mytopic,clientid=myclientid". That seems to be a
> much
> > > > > cleaner way than having all parts included in a single string
> > separated
> > > > by
> > > > > '|'.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > > > > mistry.p.bhavesh@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > HI Otis,
> > > > > >
> > > > > > What is migration path ?  If topic with special chars exists
> > already(
> > > > > > ".","-","|" etc)  in previous version of producer/consumer of
> > Kafka,
> > > > what
> > > > > > happens after the upgrade new producer or consumer (kafka
> version)
> > ?
> > > > > Also,
> > > > > > in new producer API (Kafka Trunk), does this enforce the rule
> about
> > > > > client
> > > > > > id as well ?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Bhavesh
> > > > > >
> > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > So maybe I should I should have asked the Q explicitly:
> > > > > > > Could we commit the patch from
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I
> > hope,
> > > > > it's
> > > > > > > clear what problems the current MBean names can cause?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Otis
> > > > > > > --
> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > > Management
> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > *Problem:*
> > > > > > > > Some Kafka 0.8.x MBeans have names composed of things like
> > > > <consumer
> > > > > > > > group>-<topic>-<metric name>.  Note how dashes are used as
> > > > > delimiters.
> > > > > > > >  When <consumer group> and <topic> don't contain the
> delimiter
> > > > > > character
> > > > > > > > all is good if you want to extract parts of this MBean name
> by
> > > > simply
> > > > > > > > splitting on the delimiter character.  The problem is that
> > dashes
> > > > are
> > > > > > > > allowed in topic and group names, so this splitting doesn't
> > work.
> > > > > > > > Moreover, underscores are also used as delimiters, and they
> can
> > > > also
> > > > > be
> > > > > > > > used in things like topic names.
> > > > > > > >
> > > > > > > > *Example*:
> > > > > > > > This MBean's name is composed of <consumer
> > > > > group>-<topic>-BytesPerSec:
> > > > > > > >
> > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > name="*myGroup**-myTopic**-*
> > > > > > > > BytesPerSec"
> > > > > > > >
> > > > > > > > Here we can actually split on "-" and extract all 3 parts
> from
> > > the
> > > > > > MBean
> > > > > > > > name::
> > > > > > > > * consumer group ('*myGroup*')
> > > > > > > > * topic ('*myTopic*')
> > > > > > > > * metric (‘BytesPerSec’)
> > > > > > > >
> > > > > > > > All good!
> > > > > > > >
> > > > > > > > But imagine if I named the group: *my-Group*
> > > > > > > > And if I named the topic: *my-Topic*
> > > > > > > >
> > > > > > > > Then we'd have:
> > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > name="*my-Group**-my-Topic**-*
> > > > > > > > BytesPerSec"
> > > > > > > >
> > > > > > > > Now splitting on "-" would no longer work!  To extract
> > "my-Group"
> > > > and
> > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to know the
> > > > specific
> > > > > > > group
> > > > > > > > name and topic name to look for and could not use generic
> > > approach
> > > > of
> > > > > > > just
> > > > > > > > splitting the MBean name on the delimiter.
> > > > > > > >
> > > > > > > > *Solution*:
> > > > > > > > The patch in
> https://issues.apache.org/jira/browse/KAFKA-1481
> > > > > replaces
> > > > > > > > all _ and - characters where they are used as delimiters in
> > MBean
> > > > > names
> > > > > > > > with a "|" character.  Because the "I" character is not
> allowed
> > > in
> > > > > > topic
> > > > > > > > names, consumer groups, host names, splitting on this new and
> > > > unified
> > > > > > > > delimiter works.
> > > > > > > >
> > > > > > > > I hope this explains the problem, the solution, and that this
> > can
> > > > > make
> > > > > > it
> > > > > > > > in the next 0.8.x.
> > > > > > > >
> > > > > > > > Otis
> > > > > > > > --
> > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > > > Management
> > > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Otis,

Yes, if you guys can help provide a patch in a few days, we can probably
get it to the 0.8.2 release.

Thanks,

Jun

On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi Jun,
>
> I think your MBean renaming approach will work.  I see
> https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version 0.8.2,
> but
> is not marked as a Blocker.  We'd love to get the MBeans fixed so this
> makes it in 0.8.2 release.  Do you know if this is on anyone's plate (the
> issue is currently Unassigned)?  If not, should we provide a new patch that
> uses your approach?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Otis,
> >
> > In kafka-1481, we will have to change the mbean names (at least the ones
> > with clientid and topic) anyway. Using the name/value pair in the mbean
> > name allows us to do this in a cleaner way. Yes, "," is not allowed in
> > clientid or topic.
> >
> > Bhavesh,
> >
> > Yes, I was thinking of making changes in the new metrics package.
> Something
> > like allowing the sensor names to have name/value pairs. The jmx names
> will
> > just follow accordingly. This is probably cleaner than doing the
> escaping.
> > Also, the metric names are more intuitive (otherwise, you have to know
> > which part is the clientid and which part is the topic).
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi Jun,
> > >
> > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Bhavesh,
> > > >
> > > > Yes, allowing dot in clientId and topic makes it a bit harder to
> define
> > > the
> > > > JMX bean names. I see a couple of solutions here.
> > > >
> > > > 1. Disable dot in clientId and topic names. The issue is that dot may
> > > > already be used in existing deployment.
> > > >
> > > > 2. We can represent the JMX bean name differently in the new
> producer.
> > > > Instead of
> > > >   kafka.producer.myclientid:type=mytopic
> > > > we could change it to
> > > >   kafka.producer:clientId=myclientid,topic=mytopic
> > > >
> > > > I felt that option 2 is probably better since it doesn't affect
> > existing
> > > > users.
> > > >
> > >
> > > If it doesn't affect existing users, great.
> > >
> > > If you are saying that each "piece" of MBean name could be expressed as
> > > name=value pair, with something like "," (forbidden in host names,
> topic
> > > names, client IDs, etc. I assume?) then yes, I think this would be
> easier
> > > to parse and it would be easier for people to understand what is what.
> > >
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > >
> > > >
> > > > Otis,
> > > >
> > > > We probably can also use option 2 to address KAFKA-1481. For
> > > topic/clientid
> > > > specific metrics, we could explicitly specify the metric name so that
> > it
> > > > contains "topic=mytopic,clientid=myclientid". That seems to be a much
> > > > cleaner way than having all parts included in a single string
> separated
> > > by
> > > > '|'.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > > > mistry.p.bhavesh@gmail.com>
> > > > wrote:
> > > >
> > > > > HI Otis,
> > > > >
> > > > > What is migration path ?  If topic with special chars exists
> already(
> > > > > ".","-","|" etc)  in previous version of producer/consumer of
> Kafka,
> > > what
> > > > > happens after the upgrade new producer or consumer (kafka version)
> ?
> > > > Also,
> > > > > in new producer API (Kafka Trunk), does this enforce the rule about
> > > > client
> > > > > id as well ?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bhavesh
> > > > >
> > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > > > > otis.gospodnetic@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > So maybe I should I should have asked the Q explicitly:
> > > > > > Could we commit the patch from
> > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I
> hope,
> > > > it's
> > > > > > clear what problems the current MBean names can cause?
> > > > > >
> > > > > > Thanks,
> > > > > > Otis
> > > > > > --
> > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > > > > otis.gospodnetic@gmail.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > *Problem:*
> > > > > > > Some Kafka 0.8.x MBeans have names composed of things like
> > > <consumer
> > > > > > > group>-<topic>-<metric name>.  Note how dashes are used as
> > > > delimiters.
> > > > > > >  When <consumer group> and <topic> don't contain the delimiter
> > > > > character
> > > > > > > all is good if you want to extract parts of this MBean name by
> > > simply
> > > > > > > splitting on the delimiter character.  The problem is that
> dashes
> > > are
> > > > > > > allowed in topic and group names, so this splitting doesn't
> work.
> > > > > > > Moreover, underscores are also used as delimiters, and they can
> > > also
> > > > be
> > > > > > > used in things like topic names.
> > > > > > >
> > > > > > > *Example*:
> > > > > > > This MBean's name is composed of <consumer
> > > > group>-<topic>-BytesPerSec:
> > > > > > >
> > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > name="*myGroup**-myTopic**-*
> > > > > > > BytesPerSec"
> > > > > > >
> > > > > > > Here we can actually split on "-" and extract all 3 parts from
> > the
> > > > > MBean
> > > > > > > name::
> > > > > > > * consumer group ('*myGroup*')
> > > > > > > * topic ('*myTopic*')
> > > > > > > * metric (‘BytesPerSec’)
> > > > > > >
> > > > > > > All good!
> > > > > > >
> > > > > > > But imagine if I named the group: *my-Group*
> > > > > > > And if I named the topic: *my-Topic*
> > > > > > >
> > > > > > > Then we'd have:
> > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > name="*my-Group**-my-Topic**-*
> > > > > > > BytesPerSec"
> > > > > > >
> > > > > > > Now splitting on "-" would no longer work!  To extract
> "my-Group"
> > > and
> > > > > > > "my-Topic" and "BytesPerSec" parts I would have to know the
> > > specific
> > > > > > group
> > > > > > > name and topic name to look for and could not use generic
> > approach
> > > of
> > > > > > just
> > > > > > > splitting the MBean name on the delimiter.
> > > > > > >
> > > > > > > *Solution*:
> > > > > > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481
> > > > replaces
> > > > > > > all _ and - characters where they are used as delimiters in
> MBean
> > > > names
> > > > > > > with a "|" character.  Because the "I" character is not allowed
> > in
> > > > > topic
> > > > > > > names, consumer groups, host names, splitting on this new and
> > > unified
> > > > > > > delimiter works.
> > > > > > >
> > > > > > > I hope this explains the problem, the solution, and that this
> can
> > > > make
> > > > > it
> > > > > > > in the next 0.8.x.
> > > > > > >
> > > > > > > Otis
> > > > > > > --
> > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > > Management
> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Otis Gospodnetic <ot...@gmail.com>.

Hi Jun,

I think your MBean renaming approach will work.  I see
https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version 0.8.2, but
is not marked as a Blocker.  We'd love to get the MBeans fixed so this
makes it in 0.8.2 release.  Do you know if this is on anyone's plate (the
issue is currently Unassigned)?  If not, should we provide a new patch that
uses your approach?

Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <ju...@gmail.com> wrote:

> Otis,
>
> In kafka-1481, we will have to change the mbean names (at least the ones
> with clientid and topic) anyway. Using the name/value pair in the mbean
> name allows us to do this in a cleaner way. Yes, "," is not allowed in
> clientid or topic.
>
> Bhavesh,
>
> Yes, I was thinking of making changes in the new metrics package. Something
> like allowing the sensor names to have name/value pairs. The jmx names will
> just follow accordingly. This is probably cleaner than doing the escaping.
> Also, the metric names are more intuitive (otherwise, you have to know
> which part is the clientid and which part is the topic).
>
> Thanks,
>
> Jun
>
> On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi Jun,
> >
> > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Bhavesh,
> > >
> > > Yes, allowing dot in clientId and topic makes it a bit harder to define
> > the
> > > JMX bean names. I see a couple of solutions here.
> > >
> > > 1. Disable dot in clientId and topic names. The issue is that dot may
> > > already be used in existing deployment.
> > >
> > > 2. We can represent the JMX bean name differently in the new producer.
> > > Instead of
> > >   kafka.producer.myclientid:type=mytopic
> > > we could change it to
> > >   kafka.producer:clientId=myclientid,topic=mytopic
> > >
> > > I felt that option 2 is probably better since it doesn't affect
> existing
> > > users.
> > >
> >
> > If it doesn't affect existing users, great.
> >
> > If you are saying that each "piece" of MBean name could be expressed as
> > name=value pair, with something like "," (forbidden in host names, topic
> > names, client IDs, etc. I assume?) then yes, I think this would be easier
> > to parse and it would be easier for people to understand what is what.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> >
> > >
> > > Otis,
> > >
> > > We probably can also use option 2 to address KAFKA-1481. For
> > topic/clientid
> > > specific metrics, we could explicitly specify the metric name so that
> it
> > > contains "topic=mytopic,clientid=myclientid". That seems to be a much
> > > cleaner way than having all parts included in a single string separated
> > by
> > > '|'.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > >
> > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > > mistry.p.bhavesh@gmail.com>
> > > wrote:
> > >
> > > > HI Otis,
> > > >
> > > > What is migration path ?  If topic with special chars exists already(
> > > > ".","-","|" etc)  in previous version of producer/consumer of Kafka,
> > what
> > > > happens after the upgrade new producer or consumer (kafka version) ?
> > > Also,
> > > > in new producer API (Kafka Trunk), does this enforce the rule about
> > > client
> > > > id as well ?
> > > >
> > > > Thanks,
> > > >
> > > > Bhavesh
> > > >
> > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > > > otis.gospodnetic@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > So maybe I should I should have asked the Q explicitly:
> > > > > Could we commit the patch from
> > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope,
> > > it's
> > > > > clear what problems the current MBean names can cause?
> > > > >
> > > > > Thanks,
> > > > > Otis
> > > > > --
> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > > > otis.gospodnetic@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > *Problem:*
> > > > > > Some Kafka 0.8.x MBeans have names composed of things like
> > <consumer
> > > > > > group>-<topic>-<metric name>.  Note how dashes are used as
> > > delimiters.
> > > > > >  When <consumer group> and <topic> don't contain the delimiter
> > > > character
> > > > > > all is good if you want to extract parts of this MBean name by
> > simply
> > > > > > splitting on the delimiter character.  The problem is that dashes
> > are
> > > > > > allowed in topic and group names, so this splitting doesn't work.
> > > > > > Moreover, underscores are also used as delimiters, and they can
> > also
> > > be
> > > > > > used in things like topic names.
> > > > > >
> > > > > > *Example*:
> > > > > > This MBean's name is composed of <consumer
> > > group>-<topic>-BytesPerSec:
> > > > > >
> > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > name="*myGroup**-myTopic**-*
> > > > > > BytesPerSec"
> > > > > >
> > > > > > Here we can actually split on "-" and extract all 3 parts from
> the
> > > > MBean
> > > > > > name::
> > > > > > * consumer group ('*myGroup*')
> > > > > > * topic ('*myTopic*')
> > > > > > * metric (‘BytesPerSec’)
> > > > > >
> > > > > > All good!
> > > > > >
> > > > > > But imagine if I named the group: *my-Group*
> > > > > > And if I named the topic: *my-Topic*
> > > > > >
> > > > > > Then we'd have:
> > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > name="*my-Group**-my-Topic**-*
> > > > > > BytesPerSec"
> > > > > >
> > > > > > Now splitting on "-" would no longer work!  To extract "my-Group"
> > and
> > > > > > "my-Topic" and "BytesPerSec" parts I would have to know the
> > specific
> > > > > group
> > > > > > name and topic name to look for and could not use generic
> approach
> > of
> > > > > just
> > > > > > splitting the MBean name on the delimiter.
> > > > > >
> > > > > > *Solution*:
> > > > > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481
> > > replaces
> > > > > > all _ and - characters where they are used as delimiters in MBean
> > > names
> > > > > > with a "|" character.  Because the "I" character is not allowed
> in
> > > > topic
> > > > > > names, consumer groups, host names, splitting on this new and
> > unified
> > > > > > delimiter works.
> > > > > >
> > > > > > I hope this explains the problem, the solution, and that this can
> > > make
> > > > it
> > > > > > in the next 0.8.x.
> > > > > >
> > > > > > Otis
> > > > > > --
> > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Otis,

In kafka-1481, we will have to change the mbean names (at least the ones
with clientid and topic) anyway. Using the name/value pair in the mbean
name allows us to do this in a cleaner way. Yes, "," is not allowed in
clientid or topic.

Bhavesh,

Yes, I was thinking of making changes in the new metrics package. Something
like allowing the sensor names to have name/value pairs. The jmx names will
just follow accordingly. This is probably cleaner than doing the escaping.
Also, the metric names are more intuitive (otherwise, you have to know
which part is the clientid and which part is the topic).

Thanks,

Jun

On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi Jun,
>
> On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Bhavesh,
> >
> > Yes, allowing dot in clientId and topic makes it a bit harder to define
> the
> > JMX bean names. I see a couple of solutions here.
> >
> > 1. Disable dot in clientId and topic names. The issue is that dot may
> > already be used in existing deployment.
> >
> > 2. We can represent the JMX bean name differently in the new producer.
> > Instead of
> >   kafka.producer.myclientid:type=mytopic
> > we could change it to
> >   kafka.producer:clientId=myclientid,topic=mytopic
> >
> > I felt that option 2 is probably better since it doesn't affect existing
> > users.
> >
>
> If it doesn't affect existing users, great.
>
> If you are saying that each "piece" of MBean name could be expressed as
> name=value pair, with something like "," (forbidden in host names, topic
> names, client IDs, etc. I assume?) then yes, I think this would be easier
> to parse and it would be easier for people to understand what is what.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> >
> > Otis,
> >
> > We probably can also use option 2 to address KAFKA-1481. For
> topic/clientid
> > specific metrics, we could explicitly specify the metric name so that it
> > contains "topic=mytopic,clientid=myclientid". That seems to be a much
> > cleaner way than having all parts included in a single string separated
> by
> > '|'.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> >
> > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> > mistry.p.bhavesh@gmail.com>
> > wrote:
> >
> > > HI Otis,
> > >
> > > What is migration path ?  If topic with special chars exists already(
> > > ".","-","|" etc)  in previous version of producer/consumer of Kafka,
> what
> > > happens after the upgrade new producer or consumer (kafka version) ?
> > Also,
> > > in new producer API (Kafka Trunk), does this enforce the rule about
> > client
> > > id as well ?
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > So maybe I should I should have asked the Q explicitly:
> > > > Could we commit the patch from
> > > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope,
> > it's
> > > > clear what problems the current MBean names can cause?
> > > >
> > > > Thanks,
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > > >
> > > >
> > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > > otis.gospodnetic@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > *Problem:*
> > > > > Some Kafka 0.8.x MBeans have names composed of things like
> <consumer
> > > > > group>-<topic>-<metric name>.  Note how dashes are used as
> > delimiters.
> > > > >  When <consumer group> and <topic> don't contain the delimiter
> > > character
> > > > > all is good if you want to extract parts of this MBean name by
> simply
> > > > > splitting on the delimiter character.  The problem is that dashes
> are
> > > > > allowed in topic and group names, so this splitting doesn't work.
> > > > > Moreover, underscores are also used as delimiters, and they can
> also
> > be
> > > > > used in things like topic names.
> > > > >
> > > > > *Example*:
> > > > > This MBean's name is composed of <consumer
> > group>-<topic>-BytesPerSec:
> > > > >
> > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > name="*myGroup**-myTopic**-*
> > > > > BytesPerSec"
> > > > >
> > > > > Here we can actually split on "-" and extract all 3 parts from the
> > > MBean
> > > > > name::
> > > > > * consumer group ('*myGroup*')
> > > > > * topic ('*myTopic*')
> > > > > * metric (‘BytesPerSec’)
> > > > >
> > > > > All good!
> > > > >
> > > > > But imagine if I named the group: *my-Group*
> > > > > And if I named the topic: *my-Topic*
> > > > >
> > > > > Then we'd have:
> > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > name="*my-Group**-my-Topic**-*
> > > > > BytesPerSec"
> > > > >
> > > > > Now splitting on "-" would no longer work!  To extract "my-Group"
> and
> > > > > "my-Topic" and "BytesPerSec" parts I would have to know the
> specific
> > > > group
> > > > > name and topic name to look for and could not use generic approach
> of
> > > > just
> > > > > splitting the MBean name on the delimiter.
> > > > >
> > > > > *Solution*:
> > > > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481
> > replaces
> > > > > all _ and - characters where they are used as delimiters in MBean
> > names
> > > > > with a "|" character.  Because the "I" character is not allowed in
> > > topic
> > > > > names, consumer groups, host names, splitting on this new and
> unified
> > > > > delimiter works.
> > > > >
> > > > > I hope this explains the problem, the solution, and that this can
> > make
> > > it
> > > > > in the next 0.8.x.
> > > > >
> > > > > Otis
> > > > > --
> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Otis Gospodnetic <ot...@gmail.com>.

Hi Jun,

On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com> wrote:

> Bhavesh,
>
> Yes, allowing dot in clientId and topic makes it a bit harder to define the
> JMX bean names. I see a couple of solutions here.
>
> 1. Disable dot in clientId and topic names. The issue is that dot may
> already be used in existing deployment.
>
> 2. We can represent the JMX bean name differently in the new producer.
> Instead of
>   kafka.producer.myclientid:type=mytopic
> we could change it to
>   kafka.producer:clientId=myclientid,topic=mytopic
>
> I felt that option 2 is probably better since it doesn't affect existing
> users.
>

If it doesn't affect existing users, great.

If you are saying that each "piece" of MBean name could be expressed as
name=value pair, with something like "," (forbidden in host names, topic
names, client IDs, etc. I assume?) then yes, I think this would be easier
to parse and it would be easier for people to understand what is what.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/



>
> Otis,
>
> We probably can also use option 2 to address KAFKA-1481. For topic/clientid
> specific metrics, we could explicitly specify the metric name so that it
> contains "topic=mytopic,clientid=myclientid". That seems to be a much
> cleaner way than having all parts included in a single string separated by
> '|'.
>
> Thanks,
>
> Jun
>
>
>
>
> On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com>
> wrote:
>
> > HI Otis,
> >
> > What is migration path ?  If topic with special chars exists already(
> > ".","-","|" etc)  in previous version of producer/consumer of Kafka, what
> > happens after the upgrade new producer or consumer (kafka version) ?
> Also,
> > in new producer API (Kafka Trunk), does this enforce the rule about
> client
> > id as well ?
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > So maybe I should I should have asked the Q explicitly:
> > > Could we commit the patch from
> > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope,
> it's
> > > clear what problems the current MBean names can cause?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > >
> > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > *Problem:*
> > > > Some Kafka 0.8.x MBeans have names composed of things like <consumer
> > > > group>-<topic>-<metric name>.  Note how dashes are used as
> delimiters.
> > > >  When <consumer group> and <topic> don't contain the delimiter
> > character
> > > > all is good if you want to extract parts of this MBean name by simply
> > > > splitting on the delimiter character.  The problem is that dashes are
> > > > allowed in topic and group names, so this splitting doesn't work.
> > > > Moreover, underscores are also used as delimiters, and they can also
> be
> > > > used in things like topic names.
> > > >
> > > > *Example*:
> > > > This MBean's name is composed of <consumer
> group>-<topic>-BytesPerSec:
> > > >
> > > > kafka.consumer:type="ConsumerTopicMetrics",
> > name="*myGroup**-myTopic**-*
> > > > BytesPerSec"
> > > >
> > > > Here we can actually split on "-" and extract all 3 parts from the
> > MBean
> > > > name::
> > > > * consumer group ('*myGroup*')
> > > > * topic ('*myTopic*')
> > > > * metric (‘BytesPerSec’)
> > > >
> > > > All good!
> > > >
> > > > But imagine if I named the group: *my-Group*
> > > > And if I named the topic: *my-Topic*
> > > >
> > > > Then we'd have:
> > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > name="*my-Group**-my-Topic**-*
> > > > BytesPerSec"
> > > >
> > > > Now splitting on "-" would no longer work!  To extract "my-Group" and
> > > > "my-Topic" and "BytesPerSec" parts I would have to know the specific
> > > group
> > > > name and topic name to look for and could not use generic approach of
> > > just
> > > > splitting the MBean name on the delimiter.
> > > >
> > > > *Solution*:
> > > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481
> replaces
> > > > all _ and - characters where they are used as delimiters in MBean
> names
> > > > with a "|" character.  Because the "I" character is not allowed in
> > topic
> > > > names, consumer groups, host names, splitting on this new and unified
> > > > delimiter works.
> > > >
> > > > I hope this explains the problem, the solution, and that this can
> make
> > it
> > > > in the next 0.8.x.
> > > >
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Otis Gospodnetic <ot...@gmail.com>.

Hi Jun,

On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <ju...@gmail.com> wrote:

> Bhavesh,
>
> Yes, allowing dot in clientId and topic makes it a bit harder to define the
> JMX bean names. I see a couple of solutions here.
>
> 1. Disable dot in clientId and topic names. The issue is that dot may
> already be used in existing deployment.
>
> 2. We can represent the JMX bean name differently in the new producer.
> Instead of
>   kafka.producer.myclientid:type=mytopic
> we could change it to
>   kafka.producer:clientId=myclientid,topic=mytopic
>
> I felt that option 2 is probably better since it doesn't affect existing
> users.
>

If it doesn't affect existing users, great.

If you are saying that each "piece" of MBean name could be expressed as
name=value pair, with something like "," (forbidden in host names, topic
names, client IDs, etc. I assume?) then yes, I think this would be easier
to parse and it would be easier for people to understand what is what.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/



>
> Otis,
>
> We probably can also use option 2 to address KAFKA-1481. For topic/clientid
> specific metrics, we could explicitly specify the metric name so that it
> contains "topic=mytopic,clientid=myclientid". That seems to be a much
> cleaner way than having all parts included in a single string separated by
> '|'.
>
> Thanks,
>
> Jun
>
>
>
>
> On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com>
> wrote:
>
> > HI Otis,
> >
> > What is migration path ?  If topic with special chars exists already(
> > ".","-","|" etc)  in previous version of producer/consumer of Kafka, what
> > happens after the upgrade new producer or consumer (kafka version) ?
> Also,
> > in new producer API (Kafka Trunk), does this enforce the rule about
> client
> > id as well ?
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > So maybe I should I should have asked the Q explicitly:
> > > Could we commit the patch from
> > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope,
> it's
> > > clear what problems the current MBean names can cause?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > >
> > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > *Problem:*
> > > > Some Kafka 0.8.x MBeans have names composed of things like <consumer
> > > > group>-<topic>-<metric name>.  Note how dashes are used as
> delimiters.
> > > >  When <consumer group> and <topic> don't contain the delimiter
> > character
> > > > all is good if you want to extract parts of this MBean name by simply
> > > > splitting on the delimiter character.  The problem is that dashes are
> > > > allowed in topic and group names, so this splitting doesn't work.
> > > > Moreover, underscores are also used as delimiters, and they can also
> be
> > > > used in things like topic names.
> > > >
> > > > *Example*:
> > > > This MBean's name is composed of <consumer
> group>-<topic>-BytesPerSec:
> > > >
> > > > kafka.consumer:type="ConsumerTopicMetrics",
> > name="*myGroup**-myTopic**-*
> > > > BytesPerSec"
> > > >
> > > > Here we can actually split on "-" and extract all 3 parts from the
> > MBean
> > > > name::
> > > > * consumer group ('*myGroup*')
> > > > * topic ('*myTopic*')
> > > > * metric (‘BytesPerSec’)
> > > >
> > > > All good!
> > > >
> > > > But imagine if I named the group: *my-Group*
> > > > And if I named the topic: *my-Topic*
> > > >
> > > > Then we'd have:
> > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > name="*my-Group**-my-Topic**-*
> > > > BytesPerSec"
> > > >
> > > > Now splitting on "-" would no longer work!  To extract "my-Group" and
> > > > "my-Topic" and "BytesPerSec" parts I would have to know the specific
> > > group
> > > > name and topic name to look for and could not use generic approach of
> > > just
> > > > splitting the MBean name on the delimiter.
> > > >
> > > > *Solution*:
> > > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481
> replaces
> > > > all _ and - characters where they are used as delimiters in MBean
> names
> > > > with a "|" character.  Because the "I" character is not allowed in
> > topic
> > > > names, consumer groups, host names, splitting on this new and unified
> > > > delimiter works.
> > > >
> > > > I hope this explains the problem, the solution, and that this can
> make
> > it
> > > > in the next 0.8.x.
> > > >
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Bhavesh Mistry <mi...@gmail.com>.

Sure we can do the option 2 for JMX beans.  But same solution should be
applied to producer.metrics() method for new producer.  Regardless of
metric is access (JMX or via producer), it has to be consistent naming
convention.

For example, I get following metric name when my topic is "topic.dot".  So
can we just use escape char if topic name or client.id contains Kafka
"reserved" chars (.,-,_etc).

topic.*topic.dot*.record-error-rate
topic.*topic.dot*.record-retry-rate
topic.*topic.dot*.byte-rate
topic.*topic.dot*.record-send-rate
topic.*topic.dot*.compression-rate


Thanks,

Bhavesh

On Wed, Sep 17, 2014 at 9:35 AM, Jun Rao <ju...@gmail.com> wrote:

> Bhavesh,
>
> Yes, allowing dot in clientId and topic makes it a bit harder to define the
> JMX bean names. I see a couple of solutions here.
>
> 1. Disable dot in clientId and topic names. The issue is that dot may
> already be used in existing deployment.
>
> 2. We can represent the JMX bean name differently in the new producer.
> Instead of
>   kafka.producer.myclientid:type=mytopic
> we could change it to
>   kafka.producer:clientId=myclientid,topic=mytopic
>
> I felt that option 2 is probably better since it doesn't affect existing
> users.
>
> Otis,
>
> We probably can also use option 2 to address KAFKA-1481. For topic/clientid
> specific metrics, we could explicitly specify the metric name so that it
> contains "topic=mytopic,clientid=myclientid". That seems to be a much
> cleaner way than having all parts included in a single string separated by
> '|'.
>
> Thanks,
>
> Jun
>
>
>
>
> On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com>
> wrote:
>
> > HI Otis,
> >
> > What is migration path ?  If topic with special chars exists already(
> > ".","-","|" etc)  in previous version of producer/consumer of Kafka, what
> > happens after the upgrade new producer or consumer (kafka version) ?
> Also,
> > in new producer API (Kafka Trunk), does this enforce the rule about
> client
> > id as well ?
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > So maybe I should I should have asked the Q explicitly:
> > > Could we commit the patch from
> > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope,
> it's
> > > clear what problems the current MBean names can cause?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > >
> > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > *Problem:*
> > > > Some Kafka 0.8.x MBeans have names composed of things like <consumer
> > > > group>-<topic>-<metric name>.  Note how dashes are used as
> delimiters.
> > > >  When <consumer group> and <topic> don't contain the delimiter
> > character
> > > > all is good if you want to extract parts of this MBean name by simply
> > > > splitting on the delimiter character.  The problem is that dashes are
> > > > allowed in topic and group names, so this splitting doesn't work.
> > > > Moreover, underscores are also used as delimiters, and they can also
> be
> > > > used in things like topic names.
> > > >
> > > > *Example*:
> > > > This MBean's name is composed of <consumer
> group>-<topic>-BytesPerSec:
> > > >
> > > > kafka.consumer:type="ConsumerTopicMetrics",
> > name="*myGroup**-myTopic**-*
> > > > BytesPerSec"
> > > >
> > > > Here we can actually split on "-" and extract all 3 parts from the
> > MBean
> > > > name::
> > > > * consumer group ('*myGroup*')
> > > > * topic ('*myTopic*')
> > > > * metric (‘BytesPerSec’)
> > > >
> > > > All good!
> > > >
> > > > But imagine if I named the group: *my-Group*
> > > > And if I named the topic: *my-Topic*
> > > >
> > > > Then we'd have:
> > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > name="*my-Group**-my-Topic**-*
> > > > BytesPerSec"
> > > >
> > > > Now splitting on "-" would no longer work!  To extract "my-Group" and
> > > > "my-Topic" and "BytesPerSec" parts I would have to know the specific
> > > group
> > > > name and topic name to look for and could not use generic approach of
> > > just
> > > > splitting the MBean name on the delimiter.
> > > >
> > > > *Solution*:
> > > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481
> replaces
> > > > all _ and - characters where they are used as delimiters in MBean
> names
> > > > with a "|" character.  Because the "I" character is not allowed in
> > topic
> > > > names, consumer groups, host names, splitting on this new and unified
> > > > delimiter works.
> > > >
> > > > I hope this explains the problem, the solution, and that this can
> make
> > it
> > > > in the next 0.8.x.
> > > >
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Bhavesh Mistry <mi...@gmail.com>.

Sure we can do the option 2 for JMX beans.  But same solution should be
applied to producer.metrics() method for new producer.  Regardless of
metric is access (JMX or via producer), it has to be consistent naming
convention.

For example, I get following metric name when my topic is "topic.dot".  So
can we just use escape char if topic name or client.id contains Kafka
"reserved" chars (.,-,_etc).

topic.*topic.dot*.record-error-rate
topic.*topic.dot*.record-retry-rate
topic.*topic.dot*.byte-rate
topic.*topic.dot*.record-send-rate
topic.*topic.dot*.compression-rate


Thanks,

Bhavesh

On Wed, Sep 17, 2014 at 9:35 AM, Jun Rao <ju...@gmail.com> wrote:

> Bhavesh,
>
> Yes, allowing dot in clientId and topic makes it a bit harder to define the
> JMX bean names. I see a couple of solutions here.
>
> 1. Disable dot in clientId and topic names. The issue is that dot may
> already be used in existing deployment.
>
> 2. We can represent the JMX bean name differently in the new producer.
> Instead of
>   kafka.producer.myclientid:type=mytopic
> we could change it to
>   kafka.producer:clientId=myclientid,topic=mytopic
>
> I felt that option 2 is probably better since it doesn't affect existing
> users.
>
> Otis,
>
> We probably can also use option 2 to address KAFKA-1481. For topic/clientid
> specific metrics, we could explicitly specify the metric name so that it
> contains "topic=mytopic,clientid=myclientid". That seems to be a much
> cleaner way than having all parts included in a single string separated by
> '|'.
>
> Thanks,
>
> Jun
>
>
>
>
> On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com>
> wrote:
>
> > HI Otis,
> >
> > What is migration path ?  If topic with special chars exists already(
> > ".","-","|" etc)  in previous version of producer/consumer of Kafka, what
> > happens after the upgrade new producer or consumer (kafka version) ?
> Also,
> > in new producer API (Kafka Trunk), does this enforce the rule about
> client
> > id as well ?
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > So maybe I should I should have asked the Q explicitly:
> > > Could we commit the patch from
> > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope,
> it's
> > > clear what problems the current MBean names can cause?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > >
> > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > *Problem:*
> > > > Some Kafka 0.8.x MBeans have names composed of things like <consumer
> > > > group>-<topic>-<metric name>.  Note how dashes are used as
> delimiters.
> > > >  When <consumer group> and <topic> don't contain the delimiter
> > character
> > > > all is good if you want to extract parts of this MBean name by simply
> > > > splitting on the delimiter character.  The problem is that dashes are
> > > > allowed in topic and group names, so this splitting doesn't work.
> > > > Moreover, underscores are also used as delimiters, and they can also
> be
> > > > used in things like topic names.
> > > >
> > > > *Example*:
> > > > This MBean's name is composed of <consumer
> group>-<topic>-BytesPerSec:
> > > >
> > > > kafka.consumer:type="ConsumerTopicMetrics",
> > name="*myGroup**-myTopic**-*
> > > > BytesPerSec"
> > > >
> > > > Here we can actually split on "-" and extract all 3 parts from the
> > MBean
> > > > name::
> > > > * consumer group ('*myGroup*')
> > > > * topic ('*myTopic*')
> > > > * metric (‘BytesPerSec’)
> > > >
> > > > All good!
> > > >
> > > > But imagine if I named the group: *my-Group*
> > > > And if I named the topic: *my-Topic*
> > > >
> > > > Then we'd have:
> > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > name="*my-Group**-my-Topic**-*
> > > > BytesPerSec"
> > > >
> > > > Now splitting on "-" would no longer work!  To extract "my-Group" and
> > > > "my-Topic" and "BytesPerSec" parts I would have to know the specific
> > > group
> > > > name and topic name to look for and could not use generic approach of
> > > just
> > > > splitting the MBean name on the delimiter.
> > > >
> > > > *Solution*:
> > > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481
> replaces
> > > > all _ and - characters where they are used as delimiters in MBean
> names
> > > > with a "|" character.  Because the "I" character is not allowed in
> > topic
> > > > names, consumer groups, host names, splitting on this new and unified
> > > > delimiter works.
> > > >
> > > > I hope this explains the problem, the solution, and that this can
> make
> > it
> > > > in the next 0.8.x.
> > > >
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Bhavesh,

Yes, allowing dot in clientId and topic makes it a bit harder to define the
JMX bean names. I see a couple of solutions here.

1. Disable dot in clientId and topic names. The issue is that dot may
already be used in existing deployment.

2. We can represent the JMX bean name differently in the new producer.
Instead of
  kafka.producer.myclientid:type=mytopic
we could change it to
  kafka.producer:clientId=myclientid,topic=mytopic

I felt that option 2 is probably better since it doesn't affect existing
users.

Otis,

We probably can also use option 2 to address KAFKA-1481. For topic/clientid
specific metrics, we could explicitly specify the metric name so that it
contains "topic=mytopic,clientid=myclientid". That seems to be a much
cleaner way than having all parts included in a single string separated by
'|'.

Thanks,

Jun




On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:

> HI Otis,
>
> What is migration path ?  If topic with special chars exists already(
> ".","-","|" etc)  in previous version of producer/consumer of Kafka, what
> happens after the upgrade new producer or consumer (kafka version) ?  Also,
> in new producer API (Kafka Trunk), does this enforce the rule about client
> id as well ?
>
> Thanks,
>
> Bhavesh
>
> On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi,
> >
> > So maybe I should I should have asked the Q explicitly:
> > Could we commit the patch from
> > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope, it's
> > clear what problems the current MBean names can cause?
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> >
> > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > *Problem:*
> > > Some Kafka 0.8.x MBeans have names composed of things like <consumer
> > > group>-<topic>-<metric name>.  Note how dashes are used as delimiters.
> > >  When <consumer group> and <topic> don't contain the delimiter
> character
> > > all is good if you want to extract parts of this MBean name by simply
> > > splitting on the delimiter character.  The problem is that dashes are
> > > allowed in topic and group names, so this splitting doesn't work.
> > > Moreover, underscores are also used as delimiters, and they can also be
> > > used in things like topic names.
> > >
> > > *Example*:
> > > This MBean's name is composed of <consumer group>-<topic>-BytesPerSec:
> > >
> > > kafka.consumer:type="ConsumerTopicMetrics",
> name="*myGroup**-myTopic**-*
> > > BytesPerSec"
> > >
> > > Here we can actually split on "-" and extract all 3 parts from the
> MBean
> > > name::
> > > * consumer group ('*myGroup*')
> > > * topic ('*myTopic*')
> > > * metric (‘BytesPerSec’)
> > >
> > > All good!
> > >
> > > But imagine if I named the group: *my-Group*
> > > And if I named the topic: *my-Topic*
> > >
> > > Then we'd have:
> > > kafka.consumer:type="ConsumerTopicMetrics",
> > name="*my-Group**-my-Topic**-*
> > > BytesPerSec"
> > >
> > > Now splitting on "-" would no longer work!  To extract "my-Group" and
> > > "my-Topic" and "BytesPerSec" parts I would have to know the specific
> > group
> > > name and topic name to look for and could not use generic approach of
> > just
> > > splitting the MBean name on the delimiter.
> > >
> > > *Solution*:
> > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481 replaces
> > > all _ and - characters where they are used as delimiters in MBean names
> > > with a "|" character.  Because the "I" character is not allowed in
> topic
> > > names, consumer groups, host names, splitting on this new and unified
> > > delimiter works.
> > >
> > > I hope this explains the problem, the solution, and that this can make
> it
> > > in the next 0.8.x.
> > >
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Jun Rao <ju...@gmail.com>.

Bhavesh,

Yes, allowing dot in clientId and topic makes it a bit harder to define the
JMX bean names. I see a couple of solutions here.

1. Disable dot in clientId and topic names. The issue is that dot may
already be used in existing deployment.

2. We can represent the JMX bean name differently in the new producer.
Instead of
  kafka.producer.myclientid:type=mytopic
we could change it to
  kafka.producer:clientId=myclientid,topic=mytopic

I felt that option 2 is probably better since it doesn't affect existing
users.

Otis,

We probably can also use option 2 to address KAFKA-1481. For topic/clientid
specific metrics, we could explicitly specify the metric name so that it
contains "topic=mytopic,clientid=myclientid". That seems to be a much
cleaner way than having all parts included in a single string separated by
'|'.

Thanks,

Jun




On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:

> HI Otis,
>
> What is migration path ?  If topic with special chars exists already(
> ".","-","|" etc)  in previous version of producer/consumer of Kafka, what
> happens after the upgrade new producer or consumer (kafka version) ?  Also,
> in new producer API (Kafka Trunk), does this enforce the rule about client
> id as well ?
>
> Thanks,
>
> Bhavesh
>
> On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi,
> >
> > So maybe I should I should have asked the Q explicitly:
> > Could we commit the patch from
> > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope, it's
> > clear what problems the current MBean names can cause?
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> >
> > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > *Problem:*
> > > Some Kafka 0.8.x MBeans have names composed of things like <consumer
> > > group>-<topic>-<metric name>.  Note how dashes are used as delimiters.
> > >  When <consumer group> and <topic> don't contain the delimiter
> character
> > > all is good if you want to extract parts of this MBean name by simply
> > > splitting on the delimiter character.  The problem is that dashes are
> > > allowed in topic and group names, so this splitting doesn't work.
> > > Moreover, underscores are also used as delimiters, and they can also be
> > > used in things like topic names.
> > >
> > > *Example*:
> > > This MBean's name is composed of <consumer group>-<topic>-BytesPerSec:
> > >
> > > kafka.consumer:type="ConsumerTopicMetrics",
> name="*myGroup**-myTopic**-*
> > > BytesPerSec"
> > >
> > > Here we can actually split on "-" and extract all 3 parts from the
> MBean
> > > name::
> > > * consumer group ('*myGroup*')
> > > * topic ('*myTopic*')
> > > * metric (‘BytesPerSec’)
> > >
> > > All good!
> > >
> > > But imagine if I named the group: *my-Group*
> > > And if I named the topic: *my-Topic*
> > >
> > > Then we'd have:
> > > kafka.consumer:type="ConsumerTopicMetrics",
> > name="*my-Group**-my-Topic**-*
> > > BytesPerSec"
> > >
> > > Now splitting on "-" would no longer work!  To extract "my-Group" and
> > > "my-Topic" and "BytesPerSec" parts I would have to know the specific
> > group
> > > name and topic name to look for and could not use generic approach of
> > just
> > > splitting the MBean name on the delimiter.
> > >
> > > *Solution*:
> > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481 replaces
> > > all _ and - characters where they are used as delimiters in MBean names
> > > with a "|" character.  Because the "I" character is not allowed in
> topic
> > > names, consumer groups, host names, splitting on this new and unified
> > > delimiter works.
> > >
> > > I hope this explains the problem, the solution, and that this can make
> it
> > > in the next 0.8.x.
> > >
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Bhavesh Mistry <mi...@gmail.com>.

HI Otis,

What is migration path ?  If topic with special chars exists already(
".","-","|" etc)  in previous version of producer/consumer of Kafka, what
happens after the upgrade new producer or consumer (kafka version) ?  Also,
in new producer API (Kafka Trunk), does this enforce the rule about client
id as well ?

Thanks,

Bhavesh

On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> So maybe I should I should have asked the Q explicitly:
> Could we commit the patch from
> https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope, it's
> clear what problems the current MBean names can cause?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi,
> >
> > *Problem:*
> > Some Kafka 0.8.x MBeans have names composed of things like <consumer
> > group>-<topic>-<metric name>.  Note how dashes are used as delimiters.
> >  When <consumer group> and <topic> don't contain the delimiter character
> > all is good if you want to extract parts of this MBean name by simply
> > splitting on the delimiter character.  The problem is that dashes are
> > allowed in topic and group names, so this splitting doesn't work.
> > Moreover, underscores are also used as delimiters, and they can also be
> > used in things like topic names.
> >
> > *Example*:
> > This MBean's name is composed of <consumer group>-<topic>-BytesPerSec:
> >
> > kafka.consumer:type="ConsumerTopicMetrics", name="*myGroup**-myTopic**-*
> > BytesPerSec"
> >
> > Here we can actually split on "-" and extract all 3 parts from the MBean
> > name::
> > * consumer group ('*myGroup*')
> > * topic ('*myTopic*')
> > * metric (‘BytesPerSec’)
> >
> > All good!
> >
> > But imagine if I named the group: *my-Group*
> > And if I named the topic: *my-Topic*
> >
> > Then we'd have:
> > kafka.consumer:type="ConsumerTopicMetrics",
> name="*my-Group**-my-Topic**-*
> > BytesPerSec"
> >
> > Now splitting on "-" would no longer work!  To extract "my-Group" and
> > "my-Topic" and "BytesPerSec" parts I would have to know the specific
> group
> > name and topic name to look for and could not use generic approach of
> just
> > splitting the MBean name on the delimiter.
> >
> > *Solution*:
> > The patch in https://issues.apache.org/jira/browse/KAFKA-1481 replaces
> > all _ and - characters where they are used as delimiters in MBean names
> > with a "|" character.  Because the "I" character is not allowed in topic
> > names, consumer groups, host names, splitting on this new and unified
> > delimiter works.
> >
> > I hope this explains the problem, the solution, and that this can make it
> > in the next 0.8.x.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
>

Re: MBeans, dashes, underscores, and KAFKA-1481

Posted by Bhavesh Mistry <mi...@gmail.com>.

HI Otis,

What is migration path ?  If topic with special chars exists already(
".","-","|" etc)  in previous version of producer/consumer of Kafka, what
happens after the upgrade new producer or consumer (kafka version) ?  Also,
in new producer API (Kafka Trunk), does this enforce the rule about client
id as well ?

Thanks,

Bhavesh

On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> So maybe I should I should have asked the Q explicitly:
> Could we commit the patch from
> https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope, it's
> clear what problems the current MBean names can cause?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi,
> >
> > *Problem:*
> > Some Kafka 0.8.x MBeans have names composed of things like <consumer
> > group>-<topic>-<metric name>.  Note how dashes are used as delimiters.
> >  When <consumer group> and <topic> don't contain the delimiter character
> > all is good if you want to extract parts of this MBean name by simply
> > splitting on the delimiter character.  The problem is that dashes are
> > allowed in topic and group names, so this splitting doesn't work.
> > Moreover, underscores are also used as delimiters, and they can also be
> > used in things like topic names.
> >
> > *Example*:
> > This MBean's name is composed of <consumer group>-<topic>-BytesPerSec:
> >
> > kafka.consumer:type="ConsumerTopicMetrics", name="*myGroup**-myTopic**-*
> > BytesPerSec"
> >
> > Here we can actually split on "-" and extract all 3 parts from the MBean
> > name::
> > * consumer group ('*myGroup*')
> > * topic ('*myTopic*')
> > * metric (‘BytesPerSec’)
> >
> > All good!
> >
> > But imagine if I named the group: *my-Group*
> > And if I named the topic: *my-Topic*
> >
> > Then we'd have:
> > kafka.consumer:type="ConsumerTopicMetrics",
> name="*my-Group**-my-Topic**-*
> > BytesPerSec"
> >
> > Now splitting on "-" would no longer work!  To extract "my-Group" and
> > "my-Topic" and "BytesPerSec" parts I would have to know the specific
> group
> > name and topic name to look for and could not use generic approach of
> just
> > splitting the MBean name on the delimiter.
> >
> > *Solution*:
> > The patch in https://issues.apache.org/jira/browse/KAFKA-1481 replaces
> > all _ and - characters where they are used as delimiters in MBean names
> > with a "|" character.  Because the "I" character is not allowed in topic
> > names, consumer groups, host names, splitting on this new and unified
> > delimiter works.
> >
> > I hope this explains the problem, the solution, and that this can make it
> > in the next 0.8.x.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
>