You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2014/09/09 23:07:20 UTC
Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Kafka Team,
Can you please let me know what each of following Metrics means ? Some of
them are obvious, but some are hard to understand. My Topic name is
*TOPIC_NAME*.
can we enforce a Topic Name Convention or Metric Name Convention. Because
in previous version of Kafka, we have similar issue of parsing Kafka
Metrics name with host name issue (codahale lib) . I have topic name with
“.” So, it is hard to distinguish metric name and topic. Also, when
guys get chance I would appreciate if you guys can explain metric
description on wiki so community would know what to monitor. Please see
below for full list of metrics from new producer.
Thanks,
Bhavesh
record-queue-time-avg NaN
*node-1.*request-latency-max -Infinity
record-size-max -Infinity
*node-1.*incoming-byte-rate NaN
request-size-avg NaN
*node-1.*request-latency-avg NaN
*node-2.*request-size-avg NaN
requests-in-flight 0.0
bufferpool-wait-ratio NaN
network-io-rate NaN
metadata-age 239.828
records-per-request-avg NaN
record-retry-rate NaN
buffer-total-bytes 6.7108864E7
buffer-available-bytes 6.7108864E7
topic.*TOPIC_NAME*.record-error-rate NaN
record-send-rate NaN
select-rate NaN
node-2.outgoing-byte-rate NaN
topic.*TOPIC_NAME*.record-retry-rate NaN
batch-size-max -Infinity
connection-creation-rate NaN
node-1.outgoing-byte-rate NaN
topic.*TOPIC_NAME*.byte-rate NaN
waiting-threads 0.0
batch-size-avg NaN
io-wait-ratio NaN
io-wait-time-ns-avg NaN
io-ratio NaN
topic.TOPIC_NAME.record-send-rate NaN
request-size-max -Infinity
record-size-avg NaN
request-latency-max -Infinity
node-2.request-latency-max -Infinity
record-queue-time-max -Infinity
node-2.response-rate NaN
node-1.request-rate NaN
node-1.request-size-max -Infinity
connection-count 3.0
incoming-byte-rate NaN
compression-rate-avg NaN
request-rate NaN
node-1.response-rate NaN
node-2.request-latency-avg NaN
request-latency-avg NaN
record-error-rate NaN
connection-close-rate NaN
*node-2.*request-size-max -Infinity
topic.TOPIC_NAME.compression-rate NaN
node-2.incoming-byte-rate NaN
node-1.request-size-avg NaN
io-time-ns-avg NaN
outgoing-byte-rate NaN
*node-2*.request-rate NaN
response-rate NaN
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Bhavesh Mistry <mi...@gmail.com>.
Hi Jun,
I have created this issue for tracking purpose
https://issues.apache.org/jira/browse/KAFKA-1628
Thanks,
Bhavesh
On Wed, Sep 10, 2014 at 9:06 PM, Jun Rao <ju...@gmail.com> wrote:
> Hmm, it seems that we do allow "." in the topic name. The topic name can't
> be just "." or ".." though. So, if there is a topic "test.1", we will have
> the following jmx metrics name.
>
> kafka.producer.console-producer.topic.test:type=1
>
> It should be changed to
> kafka.producer.console-producer.topic:type=test.1
>
> Could you file a jira to follow up on this?
>
> Thanks,
>
> Jun
>
> On Wed, Sep 10, 2014 at 5:56 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com>
> wrote:
>
> > I am using topic name with "." and it works with old and new
> > producers/consumers. is Kafka enforcing in code or documented
> limitation ?
> >
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Wed, Sep 10, 2014 at 3:24 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > We actually don't allow "." in the topic name. Topic name can be
> > > alpha-numeric plus "-" and "_".
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry <
> > mistry.p.bhavesh@gmail.com
> > > >
> > > wrote:
> > >
> > > > Thanks, I was using without JMX. I will go through doc. But how
> about
> > > > Topic or Metric name Topic Name Convention or Metric Name Convention
> ?
> > > The
> > > > dot notation with topic having a ".". Any future plan to enforce
> some
> > > > stand rules.
> > > >
> > > > Thanks,
> > > >
> > > > Bhavesh
> > > >
> > > > On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com>
> wrote:
> > > >
> > > > > Hi Bhavesh,
> > > > >
> > > > > Each of those JMX attributes comes with documentation. If you open
> up
> > > > > jconsole and attach to a jvm running the consumer you should be
> able
> > > > > to read the descriptions for each attribute.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> > > > > <mi...@gmail.com> wrote:
> > > > > > Kafka Team,
> > > > > >
> > > > > > Can you please let me know what each of following Metrics means ?
> > > Some
> > > > > of
> > > > > > them are obvious, but some are hard to understand. My Topic name
> is
> > > > > > *TOPIC_NAME*.
> > > > > >
> > > > > >
> > > > > >
> > > > > > can we enforce a Topic Name Convention or Metric Name Convention.
> > > > > Because
> > > > > > in previous version of Kafka, we have similar issue of parsing
> > Kafka
> > > > > > Metrics name with host name issue (codahale lib) . I have topic
> > name
> > > > > with
> > > > > > “.” So, it is hard to distinguish metric name and topic. Also,
> > > when
> > > > > > guys get chance I would appreciate if you guys can explain metric
> > > > > > description on wiki so community would know what to monitor.
> > Please
> > > > see
> > > > > > below for full list of metrics from new producer.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Bhavesh
> > > > > >
> > > > > >
> > > > > > record-queue-time-avg NaN
> > > > > > *node-1.*request-latency-max -Infinity
> > > > > > record-size-max -Infinity
> > > > > > *node-1.*incoming-byte-rate NaN
> > > > > > request-size-avg NaN
> > > > > > *node-1.*request-latency-avg NaN
> > > > > > *node-2.*request-size-avg NaN
> > > > > > requests-in-flight 0.0
> > > > > > bufferpool-wait-ratio NaN
> > > > > > network-io-rate NaN
> > > > > > metadata-age 239.828
> > > > > > records-per-request-avg NaN
> > > > > > record-retry-rate NaN
> > > > > > buffer-total-bytes 6.7108864E7
> > > > > > buffer-available-bytes 6.7108864E7
> > > > > > topic.*TOPIC_NAME*.record-error-rate NaN
> > > > > > record-send-rate NaN
> > > > > > select-rate NaN
> > > > > > node-2.outgoing-byte-rate NaN
> > > > > > topic.*TOPIC_NAME*.record-retry-rate NaN
> > > > > > batch-size-max -Infinity
> > > > > > connection-creation-rate NaN
> > > > > > node-1.outgoing-byte-rate NaN
> > > > > > topic.*TOPIC_NAME*.byte-rate NaN
> > > > > > waiting-threads 0.0
> > > > > > batch-size-avg NaN
> > > > > > io-wait-ratio NaN
> > > > > > io-wait-time-ns-avg NaN
> > > > > > io-ratio NaN
> > > > > > topic.TOPIC_NAME.record-send-rate NaN
> > > > > > request-size-max -Infinity
> > > > > > record-size-avg NaN
> > > > > > request-latency-max -Infinity
> > > > > > node-2.request-latency-max -Infinity
> > > > > > record-queue-time-max -Infinity
> > > > > > node-2.response-rate NaN
> > > > > > node-1.request-rate NaN
> > > > > > node-1.request-size-max -Infinity
> > > > > > connection-count 3.0
> > > > > > incoming-byte-rate NaN
> > > > > > compression-rate-avg NaN
> > > > > > request-rate NaN
> > > > > > node-1.response-rate NaN
> > > > > > node-2.request-latency-avg NaN
> > > > > > request-latency-avg NaN
> > > > > > record-error-rate NaN
> > > > > > connection-close-rate NaN
> > > > > > *node-2.*request-size-max -Infinity
> > > > > > topic.TOPIC_NAME.compression-rate NaN
> > > > > > node-2.incoming-byte-rate NaN
> > > > > > node-1.request-size-avg NaN
> > > > > > io-time-ns-avg NaN
> > > > > > outgoing-byte-rate NaN
> > > > > > *node-2*.request-rate NaN
> > > > > > response-rate NaN
> > > > >
> > > >
> > >
> >
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Jun Rao <ju...@gmail.com>.
Hmm, it seems that we do allow "." in the topic name. The topic name can't
be just "." or ".." though. So, if there is a topic "test.1", we will have
the following jmx metrics name.
kafka.producer.console-producer.topic.test:type=1
It should be changed to
kafka.producer.console-producer.topic:type=test.1
Could you file a jira to follow up on this?
Thanks,
Jun
On Wed, Sep 10, 2014 at 5:56 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> I am using topic name with "." and it works with old and new
> producers/consumers. is Kafka enforcing in code or documented limitation ?
>
>
> Thanks,
>
> Bhavesh
>
> On Wed, Sep 10, 2014 at 3:24 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > We actually don't allow "." in the topic name. Topic name can be
> > alpha-numeric plus "-" and "_".
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com
> > >
> > wrote:
> >
> > > Thanks, I was using without JMX. I will go through doc. But how about
> > > Topic or Metric name Topic Name Convention or Metric Name Convention ?
> > The
> > > dot notation with topic having a ".". Any future plan to enforce some
> > > stand rules.
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> > > On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com> wrote:
> > >
> > > > Hi Bhavesh,
> > > >
> > > > Each of those JMX attributes comes with documentation. If you open up
> > > > jconsole and attach to a jvm running the consumer you should be able
> > > > to read the descriptions for each attribute.
> > > >
> > > > -Jay
> > > >
> > > > On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> > > > <mi...@gmail.com> wrote:
> > > > > Kafka Team,
> > > > >
> > > > > Can you please let me know what each of following Metrics means ?
> > Some
> > > > of
> > > > > them are obvious, but some are hard to understand. My Topic name is
> > > > > *TOPIC_NAME*.
> > > > >
> > > > >
> > > > >
> > > > > can we enforce a Topic Name Convention or Metric Name Convention.
> > > > Because
> > > > > in previous version of Kafka, we have similar issue of parsing
> Kafka
> > > > > Metrics name with host name issue (codahale lib) . I have topic
> name
> > > > with
> > > > > “.” So, it is hard to distinguish metric name and topic. Also,
> > when
> > > > > guys get chance I would appreciate if you guys can explain metric
> > > > > description on wiki so community would know what to monitor.
> Please
> > > see
> > > > > below for full list of metrics from new producer.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bhavesh
> > > > >
> > > > >
> > > > > record-queue-time-avg NaN
> > > > > *node-1.*request-latency-max -Infinity
> > > > > record-size-max -Infinity
> > > > > *node-1.*incoming-byte-rate NaN
> > > > > request-size-avg NaN
> > > > > *node-1.*request-latency-avg NaN
> > > > > *node-2.*request-size-avg NaN
> > > > > requests-in-flight 0.0
> > > > > bufferpool-wait-ratio NaN
> > > > > network-io-rate NaN
> > > > > metadata-age 239.828
> > > > > records-per-request-avg NaN
> > > > > record-retry-rate NaN
> > > > > buffer-total-bytes 6.7108864E7
> > > > > buffer-available-bytes 6.7108864E7
> > > > > topic.*TOPIC_NAME*.record-error-rate NaN
> > > > > record-send-rate NaN
> > > > > select-rate NaN
> > > > > node-2.outgoing-byte-rate NaN
> > > > > topic.*TOPIC_NAME*.record-retry-rate NaN
> > > > > batch-size-max -Infinity
> > > > > connection-creation-rate NaN
> > > > > node-1.outgoing-byte-rate NaN
> > > > > topic.*TOPIC_NAME*.byte-rate NaN
> > > > > waiting-threads 0.0
> > > > > batch-size-avg NaN
> > > > > io-wait-ratio NaN
> > > > > io-wait-time-ns-avg NaN
> > > > > io-ratio NaN
> > > > > topic.TOPIC_NAME.record-send-rate NaN
> > > > > request-size-max -Infinity
> > > > > record-size-avg NaN
> > > > > request-latency-max -Infinity
> > > > > node-2.request-latency-max -Infinity
> > > > > record-queue-time-max -Infinity
> > > > > node-2.response-rate NaN
> > > > > node-1.request-rate NaN
> > > > > node-1.request-size-max -Infinity
> > > > > connection-count 3.0
> > > > > incoming-byte-rate NaN
> > > > > compression-rate-avg NaN
> > > > > request-rate NaN
> > > > > node-1.response-rate NaN
> > > > > node-2.request-latency-avg NaN
> > > > > request-latency-avg NaN
> > > > > record-error-rate NaN
> > > > > connection-close-rate NaN
> > > > > *node-2.*request-size-max -Infinity
> > > > > topic.TOPIC_NAME.compression-rate NaN
> > > > > node-2.incoming-byte-rate NaN
> > > > > node-1.request-size-avg NaN
> > > > > io-time-ns-avg NaN
> > > > > outgoing-byte-rate NaN
> > > > > *node-2*.request-rate NaN
> > > > > response-rate NaN
> > > >
> > >
> >
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Bhavesh Mistry <mi...@gmail.com>.
I am using topic name with "." and it works with old and new
producers/consumers. is Kafka enforcing in code or documented limitation ?
Thanks,
Bhavesh
On Wed, Sep 10, 2014 at 3:24 PM, Jun Rao <ju...@gmail.com> wrote:
> We actually don't allow "." in the topic name. Topic name can be
> alpha-numeric plus "-" and "_".
>
> Thanks,
>
> Jun
>
> On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com
> >
> wrote:
>
> > Thanks, I was using without JMX. I will go through doc. But how about
> > Topic or Metric name Topic Name Convention or Metric Name Convention ?
> The
> > dot notation with topic having a ".". Any future plan to enforce some
> > stand rules.
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Hi Bhavesh,
> > >
> > > Each of those JMX attributes comes with documentation. If you open up
> > > jconsole and attach to a jvm running the consumer you should be able
> > > to read the descriptions for each attribute.
> > >
> > > -Jay
> > >
> > > On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> > > <mi...@gmail.com> wrote:
> > > > Kafka Team,
> > > >
> > > > Can you please let me know what each of following Metrics means ?
> Some
> > > of
> > > > them are obvious, but some are hard to understand. My Topic name is
> > > > *TOPIC_NAME*.
> > > >
> > > >
> > > >
> > > > can we enforce a Topic Name Convention or Metric Name Convention.
> > > Because
> > > > in previous version of Kafka, we have similar issue of parsing Kafka
> > > > Metrics name with host name issue (codahale lib) . I have topic name
> > > with
> > > > “.” So, it is hard to distinguish metric name and topic. Also,
> when
> > > > guys get chance I would appreciate if you guys can explain metric
> > > > description on wiki so community would know what to monitor. Please
> > see
> > > > below for full list of metrics from new producer.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Bhavesh
> > > >
> > > >
> > > > record-queue-time-avg NaN
> > > > *node-1.*request-latency-max -Infinity
> > > > record-size-max -Infinity
> > > > *node-1.*incoming-byte-rate NaN
> > > > request-size-avg NaN
> > > > *node-1.*request-latency-avg NaN
> > > > *node-2.*request-size-avg NaN
> > > > requests-in-flight 0.0
> > > > bufferpool-wait-ratio NaN
> > > > network-io-rate NaN
> > > > metadata-age 239.828
> > > > records-per-request-avg NaN
> > > > record-retry-rate NaN
> > > > buffer-total-bytes 6.7108864E7
> > > > buffer-available-bytes 6.7108864E7
> > > > topic.*TOPIC_NAME*.record-error-rate NaN
> > > > record-send-rate NaN
> > > > select-rate NaN
> > > > node-2.outgoing-byte-rate NaN
> > > > topic.*TOPIC_NAME*.record-retry-rate NaN
> > > > batch-size-max -Infinity
> > > > connection-creation-rate NaN
> > > > node-1.outgoing-byte-rate NaN
> > > > topic.*TOPIC_NAME*.byte-rate NaN
> > > > waiting-threads 0.0
> > > > batch-size-avg NaN
> > > > io-wait-ratio NaN
> > > > io-wait-time-ns-avg NaN
> > > > io-ratio NaN
> > > > topic.TOPIC_NAME.record-send-rate NaN
> > > > request-size-max -Infinity
> > > > record-size-avg NaN
> > > > request-latency-max -Infinity
> > > > node-2.request-latency-max -Infinity
> > > > record-queue-time-max -Infinity
> > > > node-2.response-rate NaN
> > > > node-1.request-rate NaN
> > > > node-1.request-size-max -Infinity
> > > > connection-count 3.0
> > > > incoming-byte-rate NaN
> > > > compression-rate-avg NaN
> > > > request-rate NaN
> > > > node-1.response-rate NaN
> > > > node-2.request-latency-avg NaN
> > > > request-latency-avg NaN
> > > > record-error-rate NaN
> > > > connection-close-rate NaN
> > > > *node-2.*request-size-max -Infinity
> > > > topic.TOPIC_NAME.compression-rate NaN
> > > > node-2.incoming-byte-rate NaN
> > > > node-1.request-size-avg NaN
> > > > io-time-ns-avg NaN
> > > > outgoing-byte-rate NaN
> > > > *node-2*.request-rate NaN
> > > > response-rate NaN
> > >
> >
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Bhavesh,
Yes, if you look at https://issues.apache.org/jira/browse/KAFKA-1481 it
describes this sort of a problem with various delimiters used by Kafka. As
Jun suggested, I'll start a new thread and try to clearly explain the
problem.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Sep 15, 2014 at 9:41 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Jun,
>
> I also wanted to highlight the implication of having a dot "." as part of
> the topic and client.id with ".". Both topic name and client id need
> rule
> enforcement.
>
> JMX Bean names are not correct:
>
> eg:
>
> I have topics called "topic.dot" and client.id as "my.client".
>
> JMX Bean name is:
>
> kafka.producer.*topic*:type=*dot*
> kafka.producer.*my.client*.topic.topic:type=dot
>
>
> With this developing a generic tool to flush the JMX values to some
> reporting tool for monitoring will be very hard to develop. Please let me
> know your recommendation. We had similar issues in Kafka old producer.
>
> At my company, we have convention to use "." to separation out
> functionality of website. eg "search.log" "checkout.log" etc. Please
> define/enforce rule for topic name and client.id. (Also, you might have
> to
> define a process of migrating topic name and client.id with new rules that
> you enforce with new producer).
>
> Thanks,
>
> Bhavesh
>
> On Mon, Sep 15, 2014 at 5:11 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Yes, that description is not precise. We do allow dots in general.
> However,
> > a topic can't be just "." or "..".
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Sep 15, 2014 at 9:31 AM, Michael G. Noll <
> > michael+storm@michael-noll.com> wrote:
> >
> > > Ah -- I only tested with dots "."
> > >
> > > However, haven't you said earlier in this thread that only "-" and "_"
> > are
> > > allowed as special chars? This would exclude dots, although in the
> error
> > > message below they (dots) are said to be fine. Is this a bug or a
> > feature?
> > > ;-)
> > >
> > > --Michael
> > >
> > >
> > >
> > > > On 14.09.2014, at 20:41, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > > Actually, we do give you an error if an invalid topic is created
> > through
> > > > CLI. When we add a create topic api, we can return the correct error
> > code
> > > > too.
> > > >
> > > > bin/kafka-topics.sh --zookeeper localhost:2181 --create
> --topic="te+dd"
> > > > --partitions 1 --replication-factor 1
> > > >
> > > > Error while executing topic command topic name te+dd is illegal,
> > > contains a
> > > > character other than ASCII alphanumerics, '.', '_' and '-'
> > > >
> > > > kafka.common.InvalidTopicException: topic name te+dd is illegal,
> > > contains a
> > > > character other than ASCII alphanumerics, '.', '_' and '-'
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Sun, Sep 14, 2014 at 1:46 AM, Michael G. Noll <
> > > michael@michael-noll.com>
> > > > wrote:
> > > >
> > > >> Wouldn't it be helpful to throw an error or a warning if the user
> > > >> tries to create a topic with an invalid name? Currently neither the
> > > >> API nor the CLI tools inform you that you are naming a topic in a
> way
> > > >> you shouldn't.
> > > >>
> > > >> And as Otis pointed out elsewhere in this thread this ties back into
> > > >> the JMX/MBean issues related the usage of "-"/dashes and
> > > >> "_"/underscores, which are unfortunately the only non-alphanumeric
> > > >> characters that are at your disposal to add reasonable "structure"
> to
> > > >> your Kafka topic names. (
> > > https://issues.apache.org/jira/browse/KAFKA-1481)
> > > >>
> > > >> Do you mind sharing your topic naming convention at LinkedIn?
> > > >>
> > > >> --Michael
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> On 11.09.2014 00:24, Jun Rao wrote:
> > > >>> We actually don't allow "." in the topic name. Topic name can be
> > > >>> alpha-numeric plus "-" and "_".
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>> On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry
> > > >>> <mi...@gmail.com> wrote:
> > > >>>
> > > >>>> Thanks, I was using without JMX. I will go through doc. But how
> > > >>>> about Topic or Metric name Topic Name Convention or Metric Name
> > > >>>> Convention ? The dot notation with topic having a ".". Any
> > > >>>> future plan to enforce some stand rules.
> > > >>>>
> > > >>>> Thanks,
> > > >>>>
> > > >>>> Bhavesh
> > > >>>>
> > > >>>> On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi Bhavesh,
> > > >>>>>
> > > >>>>> Each of those JMX attributes comes with documentation. If you
> > > >>>>> open up jconsole and attach to a jvm running the consumer you
> > > >>>>> should be able to read the descriptions for each attribute.
> > > >>>>>
> > > >>>>> -Jay
> > > >>>>>
> > > >>>>> On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> > > >>>>> <mi...@gmail.com> wrote:
> > > >>>>>> Kafka Team,
> > > >>>>>>
> > > >>>>>> Can you please let me know what each of following Metrics
> > > >>>>>> means ? Some
> > > >>>>> of
> > > >>>>>> them are obvious, but some are hard to understand. My Topic
> > > >>>>>> name is *TOPIC_NAME*.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> can we enforce a Topic Name Convention or Metric Name
> > > >>>>>> Convention.
> > > >>>>> Because
> > > >>>>>> in previous version of Kafka, we have similar issue of
> > > >>>>>> parsing Kafka Metrics name with host name issue (codahale
> > > >>>>>> lib) . I have topic name
> > > >>>>> with
> > > >>>>>> “.” So, it is hard to distinguish metric name and topic.
> > > >>>>>> Also, when guys get chance I would appreciate if you guys
> > > >>>>>> can explain metric description on wiki so community would
> > > >>>>>> know what to monitor. Please
> > > >>>> see
> > > >>>>>> below for full list of metrics from new producer.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>>
> > > >>>>>> Bhavesh
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> record-queue-time-avg NaN *node-1.*request-latency-max
> > > >>>>>> -Infinity record-size-max -Infinity
> > > >>>>>> *node-1.*incoming-byte-rate NaN request-size-avg NaN
> > > >>>>>> *node-1.*request-latency-avg NaN *node-2.*request-size-avg
> > > >>>>>> NaN requests-in-flight 0.0 bufferpool-wait-ratio NaN
> > > >>>>>> network-io-rate NaN metadata-age 239.828
> > > >>>>>> records-per-request-avg NaN record-retry-rate NaN
> > > >>>>>> buffer-total-bytes 6.7108864E7 buffer-available-bytes
> > > >>>>>> 6.7108864E7 topic.*TOPIC_NAME*.record-error-rate NaN
> > > >>>>>> record-send-rate NaN select-rate NaN
> > > >>>>>> node-2.outgoing-byte-rate NaN
> > > >>>>>> topic.*TOPIC_NAME*.record-retry-rate NaN batch-size-max
> > > >>>>>> -Infinity connection-creation-rate NaN
> > > >>>>>> node-1.outgoing-byte-rate NaN topic.*TOPIC_NAME*.byte-rate
> > > >>>>>> NaN waiting-threads 0.0 batch-size-avg NaN io-wait-ratio NaN
> > > >>>>>> io-wait-time-ns-avg NaN io-ratio NaN
> > > >>>>>> topic.TOPIC_NAME.record-send-rate NaN request-size-max
> > > >>>>>> -Infinity record-size-avg NaN request-latency-max -Infinity
> > > >>>>>> node-2.request-latency-max -Infinity record-queue-time-max
> > > >>>>>> -Infinity node-2.response-rate NaN node-1.request-rate NaN
> > > >>>>>> node-1.request-size-max -Infinity connection-count 3.0
> > > >>>>>> incoming-byte-rate NaN compression-rate-avg NaN request-rate
> > > >>>>>> NaN node-1.response-rate NaN node-2.request-latency-avg NaN
> > > >>>>>> request-latency-avg NaN record-error-rate NaN
> > > >>>>>> connection-close-rate NaN *node-2.*request-size-max
> > > >>>>>> -Infinity topic.TOPIC_NAME.compression-rate NaN
> > > >>>>>> node-2.incoming-byte-rate NaN node-1.request-size-avg NaN
> > > >>>>>> io-time-ns-avg NaN outgoing-byte-rate NaN
> > > >>>>>> *node-2*.request-rate NaN response-rate NaN
> > > >>
> > >
> >
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Bhavesh Mistry <mi...@gmail.com>.
Hi Jun,
I also wanted to highlight the implication of having a dot "." as part of
the topic and client.id with ".". Both topic name and client id need rule
enforcement.
JMX Bean names are not correct:
eg:
I have topics called "topic.dot" and client.id as "my.client".
JMX Bean name is:
kafka.producer.*topic*:type=*dot*
kafka.producer.*my.client*.topic.topic:type=dot
With this developing a generic tool to flush the JMX values to some
reporting tool for monitoring will be very hard to develop. Please let me
know your recommendation. We had similar issues in Kafka old producer.
At my company, we have convention to use "." to separation out
functionality of website. eg "search.log" "checkout.log" etc. Please
define/enforce rule for topic name and client.id. (Also, you might have to
define a process of migrating topic name and client.id with new rules that
you enforce with new producer).
Thanks,
Bhavesh
On Mon, Sep 15, 2014 at 5:11 PM, Jun Rao <ju...@gmail.com> wrote:
> Yes, that description is not precise. We do allow dots in general. However,
> a topic can't be just "." or "..".
>
> Thanks,
>
> Jun
>
> On Mon, Sep 15, 2014 at 9:31 AM, Michael G. Noll <
> michael+storm@michael-noll.com> wrote:
>
> > Ah -- I only tested with dots "."
> >
> > However, haven't you said earlier in this thread that only "-" and "_"
> are
> > allowed as special chars? This would exclude dots, although in the error
> > message below they (dots) are said to be fine. Is this a bug or a
> feature?
> > ;-)
> >
> > --Michael
> >
> >
> >
> > > On 14.09.2014, at 20:41, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > Actually, we do give you an error if an invalid topic is created
> through
> > > CLI. When we add a create topic api, we can return the correct error
> code
> > > too.
> > >
> > > bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic="te+dd"
> > > --partitions 1 --replication-factor 1
> > >
> > > Error while executing topic command topic name te+dd is illegal,
> > contains a
> > > character other than ASCII alphanumerics, '.', '_' and '-'
> > >
> > > kafka.common.InvalidTopicException: topic name te+dd is illegal,
> > contains a
> > > character other than ASCII alphanumerics, '.', '_' and '-'
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sun, Sep 14, 2014 at 1:46 AM, Michael G. Noll <
> > michael@michael-noll.com>
> > > wrote:
> > >
> > >> Wouldn't it be helpful to throw an error or a warning if the user
> > >> tries to create a topic with an invalid name? Currently neither the
> > >> API nor the CLI tools inform you that you are naming a topic in a way
> > >> you shouldn't.
> > >>
> > >> And as Otis pointed out elsewhere in this thread this ties back into
> > >> the JMX/MBean issues related the usage of "-"/dashes and
> > >> "_"/underscores, which are unfortunately the only non-alphanumeric
> > >> characters that are at your disposal to add reasonable "structure" to
> > >> your Kafka topic names. (
> > https://issues.apache.org/jira/browse/KAFKA-1481)
> > >>
> > >> Do you mind sharing your topic naming convention at LinkedIn?
> > >>
> > >> --Michael
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>> On 11.09.2014 00:24, Jun Rao wrote:
> > >>> We actually don't allow "." in the topic name. Topic name can be
> > >>> alpha-numeric plus "-" and "_".
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jun
> > >>>
> > >>> On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry
> > >>> <mi...@gmail.com> wrote:
> > >>>
> > >>>> Thanks, I was using without JMX. I will go through doc. But how
> > >>>> about Topic or Metric name Topic Name Convention or Metric Name
> > >>>> Convention ? The dot notation with topic having a ".". Any
> > >>>> future plan to enforce some stand rules.
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Bhavesh
> > >>>>
> > >>>> On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Bhavesh,
> > >>>>>
> > >>>>> Each of those JMX attributes comes with documentation. If you
> > >>>>> open up jconsole and attach to a jvm running the consumer you
> > >>>>> should be able to read the descriptions for each attribute.
> > >>>>>
> > >>>>> -Jay
> > >>>>>
> > >>>>> On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> > >>>>> <mi...@gmail.com> wrote:
> > >>>>>> Kafka Team,
> > >>>>>>
> > >>>>>> Can you please let me know what each of following Metrics
> > >>>>>> means ? Some
> > >>>>> of
> > >>>>>> them are obvious, but some are hard to understand. My Topic
> > >>>>>> name is *TOPIC_NAME*.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> can we enforce a Topic Name Convention or Metric Name
> > >>>>>> Convention.
> > >>>>> Because
> > >>>>>> in previous version of Kafka, we have similar issue of
> > >>>>>> parsing Kafka Metrics name with host name issue (codahale
> > >>>>>> lib) . I have topic name
> > >>>>> with
> > >>>>>> “.” So, it is hard to distinguish metric name and topic.
> > >>>>>> Also, when guys get chance I would appreciate if you guys
> > >>>>>> can explain metric description on wiki so community would
> > >>>>>> know what to monitor. Please
> > >>>> see
> > >>>>>> below for full list of metrics from new producer.
> > >>>>>>
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Bhavesh
> > >>>>>>
> > >>>>>>
> > >>>>>> record-queue-time-avg NaN *node-1.*request-latency-max
> > >>>>>> -Infinity record-size-max -Infinity
> > >>>>>> *node-1.*incoming-byte-rate NaN request-size-avg NaN
> > >>>>>> *node-1.*request-latency-avg NaN *node-2.*request-size-avg
> > >>>>>> NaN requests-in-flight 0.0 bufferpool-wait-ratio NaN
> > >>>>>> network-io-rate NaN metadata-age 239.828
> > >>>>>> records-per-request-avg NaN record-retry-rate NaN
> > >>>>>> buffer-total-bytes 6.7108864E7 buffer-available-bytes
> > >>>>>> 6.7108864E7 topic.*TOPIC_NAME*.record-error-rate NaN
> > >>>>>> record-send-rate NaN select-rate NaN
> > >>>>>> node-2.outgoing-byte-rate NaN
> > >>>>>> topic.*TOPIC_NAME*.record-retry-rate NaN batch-size-max
> > >>>>>> -Infinity connection-creation-rate NaN
> > >>>>>> node-1.outgoing-byte-rate NaN topic.*TOPIC_NAME*.byte-rate
> > >>>>>> NaN waiting-threads 0.0 batch-size-avg NaN io-wait-ratio NaN
> > >>>>>> io-wait-time-ns-avg NaN io-ratio NaN
> > >>>>>> topic.TOPIC_NAME.record-send-rate NaN request-size-max
> > >>>>>> -Infinity record-size-avg NaN request-latency-max -Infinity
> > >>>>>> node-2.request-latency-max -Infinity record-queue-time-max
> > >>>>>> -Infinity node-2.response-rate NaN node-1.request-rate NaN
> > >>>>>> node-1.request-size-max -Infinity connection-count 3.0
> > >>>>>> incoming-byte-rate NaN compression-rate-avg NaN request-rate
> > >>>>>> NaN node-1.response-rate NaN node-2.request-latency-avg NaN
> > >>>>>> request-latency-avg NaN record-error-rate NaN
> > >>>>>> connection-close-rate NaN *node-2.*request-size-max
> > >>>>>> -Infinity topic.TOPIC_NAME.compression-rate NaN
> > >>>>>> node-2.incoming-byte-rate NaN node-1.request-size-avg NaN
> > >>>>>> io-time-ns-avg NaN outgoing-byte-rate NaN
> > >>>>>> *node-2*.request-rate NaN response-rate NaN
> > >>
> >
>
Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk
Posted by "Michael G. Noll" <mi...@michael-noll.com>.
Many thanks for the clarification, Jun!
Michael
> On 16.09.2014, at 02:11, Jun Rao <ju...@gmail.com> wrote:
>
> Yes, that description is not precise. We do allow dots in general. However,
> a topic can't be just "." or "..".
>
> Thanks,
>
> Jun
>
> On Mon, Sep 15, 2014 at 9:31 AM, Michael G. Noll <
> michael+storm@michael-noll.com> wrote:
>
>> Ah -- I only tested with dots "."
>>
>> However, haven't you said earlier in this thread that only "-" and "_" are
>> allowed as special chars? This would exclude dots, although in the error
>> message below they (dots) are said to be fine. Is this a bug or a feature?
>> ;-)
>>
>> --Michael
>>
>>
>>
>>> On 14.09.2014, at 20:41, Jun Rao <ju...@gmail.com> wrote:
>>>
>>> Actually, we do give you an error if an invalid topic is created through
>>> CLI. When we add a create topic api, we can return the correct error code
>>> too.
>>>
>>> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic="te+dd"
>>> --partitions 1 --replication-factor 1
>>>
>>> Error while executing topic command topic name te+dd is illegal,
>> contains a
>>> character other than ASCII alphanumerics, '.', '_' and '-'
>>>
>>> kafka.common.InvalidTopicException: topic name te+dd is illegal,
>> contains a
>>> character other than ASCII alphanumerics, '.', '_' and '-'
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Sun, Sep 14, 2014 at 1:46 AM, Michael G. Noll <
>> michael@michael-noll.com>
>>> wrote:
>>>
>>>> Wouldn't it be helpful to throw an error or a warning if the user
>>>> tries to create a topic with an invalid name? Currently neither the
>>>> API nor the CLI tools inform you that you are naming a topic in a way
>>>> you shouldn't.
>>>>
>>>> And as Otis pointed out elsewhere in this thread this ties back into
>>>> the JMX/MBean issues related the usage of "-"/dashes and
>>>> "_"/underscores, which are unfortunately the only non-alphanumeric
>>>> characters that are at your disposal to add reasonable "structure" to
>>>> your Kafka topic names. (
>> https://issues.apache.org/jira/browse/KAFKA-1481)
>>>>
>>>> Do you mind sharing your topic naming convention at LinkedIn?
>>>>
>>>> --Michael
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On 11.09.2014 00:24, Jun Rao wrote:
>>>>> We actually don't allow "." in the topic name. Topic name can be
>>>>> alpha-numeric plus "-" and "_".
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry
>>>>> <mi...@gmail.com> wrote:
>>>>>
>>>>>> Thanks, I was using without JMX. I will go through doc. But how
>>>>>> about Topic or Metric name Topic Name Convention or Metric Name
>>>>>> Convention ? The dot notation with topic having a ".". Any
>>>>>> future plan to enforce some stand rules.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Bhavesh
>>>>>>
>>>>>> On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Bhavesh,
>>>>>>>
>>>>>>> Each of those JMX attributes comes with documentation. If you
>>>>>>> open up jconsole and attach to a jvm running the consumer you
>>>>>>> should be able to read the descriptions for each attribute.
>>>>>>>
>>>>>>> -Jay
>>>>>>>
>>>>>>> On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
>>>>>>> <mi...@gmail.com> wrote:
>>>>>>>> Kafka Team,
>>>>>>>>
>>>>>>>> Can you please let me know what each of following Metrics
>>>>>>>> means ? Some
>>>>>>> of
>>>>>>>> them are obvious, but some are hard to understand. My Topic
>>>>>>>> name is *TOPIC_NAME*.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> can we enforce a Topic Name Convention or Metric Name
>>>>>>>> Convention.
>>>>>>> Because
>>>>>>>> in previous version of Kafka, we have similar issue of
>>>>>>>> parsing Kafka Metrics name with host name issue (codahale
>>>>>>>> lib) . I have topic name
>>>>>>> with
>>>>>>>> “.” So, it is hard to distinguish metric name and topic.
>>>>>>>> Also, when guys get chance I would appreciate if you guys
>>>>>>>> can explain metric description on wiki so community would
>>>>>>>> know what to monitor. Please
>>>>>> see
>>>>>>>> below for full list of metrics from new producer.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Bhavesh
>>>>>>>>
>>>>>>>>
>>>>>>>> record-queue-time-avg NaN *node-1.*request-latency-max
>>>>>>>> -Infinity record-size-max -Infinity
>>>>>>>> *node-1.*incoming-byte-rate NaN request-size-avg NaN
>>>>>>>> *node-1.*request-latency-avg NaN *node-2.*request-size-avg
>>>>>>>> NaN requests-in-flight 0.0 bufferpool-wait-ratio NaN
>>>>>>>> network-io-rate NaN metadata-age 239.828
>>>>>>>> records-per-request-avg NaN record-retry-rate NaN
>>>>>>>> buffer-total-bytes 6.7108864E7 buffer-available-bytes
>>>>>>>> 6.7108864E7 topic.*TOPIC_NAME*.record-error-rate NaN
>>>>>>>> record-send-rate NaN select-rate NaN
>>>>>>>> node-2.outgoing-byte-rate NaN
>>>>>>>> topic.*TOPIC_NAME*.record-retry-rate NaN batch-size-max
>>>>>>>> -Infinity connection-creation-rate NaN
>>>>>>>> node-1.outgoing-byte-rate NaN topic.*TOPIC_NAME*.byte-rate
>>>>>>>> NaN waiting-threads 0.0 batch-size-avg NaN io-wait-ratio NaN
>>>>>>>> io-wait-time-ns-avg NaN io-ratio NaN
>>>>>>>> topic.TOPIC_NAME.record-send-rate NaN request-size-max
>>>>>>>> -Infinity record-size-avg NaN request-latency-max -Infinity
>>>>>>>> node-2.request-latency-max -Infinity record-queue-time-max
>>>>>>>> -Infinity node-2.response-rate NaN node-1.request-rate NaN
>>>>>>>> node-1.request-size-max -Infinity connection-count 3.0
>>>>>>>> incoming-byte-rate NaN compression-rate-avg NaN request-rate
>>>>>>>> NaN node-1.response-rate NaN node-2.request-latency-avg NaN
>>>>>>>> request-latency-avg NaN record-error-rate NaN
>>>>>>>> connection-close-rate NaN *node-2.*request-size-max
>>>>>>>> -Infinity topic.TOPIC_NAME.compression-rate NaN
>>>>>>>> node-2.incoming-byte-rate NaN node-1.request-size-avg NaN
>>>>>>>> io-time-ns-avg NaN outgoing-byte-rate NaN
>>>>>>>> *node-2*.request-rate NaN response-rate NaN
>>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Jun Rao <ju...@gmail.com>.
Yes, that description is not precise. We do allow dots in general. However,
a topic can't be just "." or "..".
Thanks,
Jun
On Mon, Sep 15, 2014 at 9:31 AM, Michael G. Noll <
michael+storm@michael-noll.com> wrote:
> Ah -- I only tested with dots "."
>
> However, haven't you said earlier in this thread that only "-" and "_" are
> allowed as special chars? This would exclude dots, although in the error
> message below they (dots) are said to be fine. Is this a bug or a feature?
> ;-)
>
> --Michael
>
>
>
> > On 14.09.2014, at 20:41, Jun Rao <ju...@gmail.com> wrote:
> >
> > Actually, we do give you an error if an invalid topic is created through
> > CLI. When we add a create topic api, we can return the correct error code
> > too.
> >
> > bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic="te+dd"
> > --partitions 1 --replication-factor 1
> >
> > Error while executing topic command topic name te+dd is illegal,
> contains a
> > character other than ASCII alphanumerics, '.', '_' and '-'
> >
> > kafka.common.InvalidTopicException: topic name te+dd is illegal,
> contains a
> > character other than ASCII alphanumerics, '.', '_' and '-'
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Sep 14, 2014 at 1:46 AM, Michael G. Noll <
> michael@michael-noll.com>
> > wrote:
> >
> >> Wouldn't it be helpful to throw an error or a warning if the user
> >> tries to create a topic with an invalid name? Currently neither the
> >> API nor the CLI tools inform you that you are naming a topic in a way
> >> you shouldn't.
> >>
> >> And as Otis pointed out elsewhere in this thread this ties back into
> >> the JMX/MBean issues related the usage of "-"/dashes and
> >> "_"/underscores, which are unfortunately the only non-alphanumeric
> >> characters that are at your disposal to add reasonable "structure" to
> >> your Kafka topic names. (
> https://issues.apache.org/jira/browse/KAFKA-1481)
> >>
> >> Do you mind sharing your topic naming convention at LinkedIn?
> >>
> >> --Michael
> >>
> >>
> >>
> >>
> >>
> >>> On 11.09.2014 00:24, Jun Rao wrote:
> >>> We actually don't allow "." in the topic name. Topic name can be
> >>> alpha-numeric plus "-" and "_".
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry
> >>> <mi...@gmail.com> wrote:
> >>>
> >>>> Thanks, I was using without JMX. I will go through doc. But how
> >>>> about Topic or Metric name Topic Name Convention or Metric Name
> >>>> Convention ? The dot notation with topic having a ".". Any
> >>>> future plan to enforce some stand rules.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Bhavesh
> >>>>
> >>>> On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Bhavesh,
> >>>>>
> >>>>> Each of those JMX attributes comes with documentation. If you
> >>>>> open up jconsole and attach to a jvm running the consumer you
> >>>>> should be able to read the descriptions for each attribute.
> >>>>>
> >>>>> -Jay
> >>>>>
> >>>>> On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> >>>>> <mi...@gmail.com> wrote:
> >>>>>> Kafka Team,
> >>>>>>
> >>>>>> Can you please let me know what each of following Metrics
> >>>>>> means ? Some
> >>>>> of
> >>>>>> them are obvious, but some are hard to understand. My Topic
> >>>>>> name is *TOPIC_NAME*.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> can we enforce a Topic Name Convention or Metric Name
> >>>>>> Convention.
> >>>>> Because
> >>>>>> in previous version of Kafka, we have similar issue of
> >>>>>> parsing Kafka Metrics name with host name issue (codahale
> >>>>>> lib) . I have topic name
> >>>>> with
> >>>>>> “.” So, it is hard to distinguish metric name and topic.
> >>>>>> Also, when guys get chance I would appreciate if you guys
> >>>>>> can explain metric description on wiki so community would
> >>>>>> know what to monitor. Please
> >>>> see
> >>>>>> below for full list of metrics from new producer.
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Bhavesh
> >>>>>>
> >>>>>>
> >>>>>> record-queue-time-avg NaN *node-1.*request-latency-max
> >>>>>> -Infinity record-size-max -Infinity
> >>>>>> *node-1.*incoming-byte-rate NaN request-size-avg NaN
> >>>>>> *node-1.*request-latency-avg NaN *node-2.*request-size-avg
> >>>>>> NaN requests-in-flight 0.0 bufferpool-wait-ratio NaN
> >>>>>> network-io-rate NaN metadata-age 239.828
> >>>>>> records-per-request-avg NaN record-retry-rate NaN
> >>>>>> buffer-total-bytes 6.7108864E7 buffer-available-bytes
> >>>>>> 6.7108864E7 topic.*TOPIC_NAME*.record-error-rate NaN
> >>>>>> record-send-rate NaN select-rate NaN
> >>>>>> node-2.outgoing-byte-rate NaN
> >>>>>> topic.*TOPIC_NAME*.record-retry-rate NaN batch-size-max
> >>>>>> -Infinity connection-creation-rate NaN
> >>>>>> node-1.outgoing-byte-rate NaN topic.*TOPIC_NAME*.byte-rate
> >>>>>> NaN waiting-threads 0.0 batch-size-avg NaN io-wait-ratio NaN
> >>>>>> io-wait-time-ns-avg NaN io-ratio NaN
> >>>>>> topic.TOPIC_NAME.record-send-rate NaN request-size-max
> >>>>>> -Infinity record-size-avg NaN request-latency-max -Infinity
> >>>>>> node-2.request-latency-max -Infinity record-queue-time-max
> >>>>>> -Infinity node-2.response-rate NaN node-1.request-rate NaN
> >>>>>> node-1.request-size-max -Infinity connection-count 3.0
> >>>>>> incoming-byte-rate NaN compression-rate-avg NaN request-rate
> >>>>>> NaN node-1.response-rate NaN node-2.request-latency-avg NaN
> >>>>>> request-latency-avg NaN record-error-rate NaN
> >>>>>> connection-close-rate NaN *node-2.*request-size-max
> >>>>>> -Infinity topic.TOPIC_NAME.compression-rate NaN
> >>>>>> node-2.incoming-byte-rate NaN node-1.request-size-avg NaN
> >>>>>> io-time-ns-avg NaN outgoing-byte-rate NaN
> >>>>>> *node-2*.request-rate NaN response-rate NaN
> >>
>
Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk
Posted by "Michael G. Noll" <mi...@michael-noll.com>.
Ah -- I only tested with dots "."
However, haven't you said earlier in this thread that only "-" and "_" are allowed as special chars? This would exclude dots, although in the error message below they (dots) are said to be fine. Is this a bug or a feature? ;-)
--Michael
> On 14.09.2014, at 20:41, Jun Rao <ju...@gmail.com> wrote:
>
> Actually, we do give you an error if an invalid topic is created through
> CLI. When we add a create topic api, we can return the correct error code
> too.
>
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic="te+dd"
> --partitions 1 --replication-factor 1
>
> Error while executing topic command topic name te+dd is illegal, contains a
> character other than ASCII alphanumerics, '.', '_' and '-'
>
> kafka.common.InvalidTopicException: topic name te+dd is illegal, contains a
> character other than ASCII alphanumerics, '.', '_' and '-'
>
> Thanks,
>
> Jun
>
> On Sun, Sep 14, 2014 at 1:46 AM, Michael G. Noll <mi...@michael-noll.com>
> wrote:
>
>> Wouldn't it be helpful to throw an error or a warning if the user
>> tries to create a topic with an invalid name? Currently neither the
>> API nor the CLI tools inform you that you are naming a topic in a way
>> you shouldn't.
>>
>> And as Otis pointed out elsewhere in this thread this ties back into
>> the JMX/MBean issues related the usage of "-"/dashes and
>> "_"/underscores, which are unfortunately the only non-alphanumeric
>> characters that are at your disposal to add reasonable "structure" to
>> your Kafka topic names. (https://issues.apache.org/jira/browse/KAFKA-1481)
>>
>> Do you mind sharing your topic naming convention at LinkedIn?
>>
>> --Michael
>>
>>
>>
>>
>>
>>> On 11.09.2014 00:24, Jun Rao wrote:
>>> We actually don't allow "." in the topic name. Topic name can be
>>> alpha-numeric plus "-" and "_".
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry
>>> <mi...@gmail.com> wrote:
>>>
>>>> Thanks, I was using without JMX. I will go through doc. But how
>>>> about Topic or Metric name Topic Name Convention or Metric Name
>>>> Convention ? The dot notation with topic having a ".". Any
>>>> future plan to enforce some stand rules.
>>>>
>>>> Thanks,
>>>>
>>>> Bhavesh
>>>>
>>>> On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Bhavesh,
>>>>>
>>>>> Each of those JMX attributes comes with documentation. If you
>>>>> open up jconsole and attach to a jvm running the consumer you
>>>>> should be able to read the descriptions for each attribute.
>>>>>
>>>>> -Jay
>>>>>
>>>>> On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
>>>>> <mi...@gmail.com> wrote:
>>>>>> Kafka Team,
>>>>>>
>>>>>> Can you please let me know what each of following Metrics
>>>>>> means ? Some
>>>>> of
>>>>>> them are obvious, but some are hard to understand. My Topic
>>>>>> name is *TOPIC_NAME*.
>>>>>>
>>>>>>
>>>>>>
>>>>>> can we enforce a Topic Name Convention or Metric Name
>>>>>> Convention.
>>>>> Because
>>>>>> in previous version of Kafka, we have similar issue of
>>>>>> parsing Kafka Metrics name with host name issue (codahale
>>>>>> lib) . I have topic name
>>>>> with
>>>>>> “.” So, it is hard to distinguish metric name and topic.
>>>>>> Also, when guys get chance I would appreciate if you guys
>>>>>> can explain metric description on wiki so community would
>>>>>> know what to monitor. Please
>>>> see
>>>>>> below for full list of metrics from new producer.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Bhavesh
>>>>>>
>>>>>>
>>>>>> record-queue-time-avg NaN *node-1.*request-latency-max
>>>>>> -Infinity record-size-max -Infinity
>>>>>> *node-1.*incoming-byte-rate NaN request-size-avg NaN
>>>>>> *node-1.*request-latency-avg NaN *node-2.*request-size-avg
>>>>>> NaN requests-in-flight 0.0 bufferpool-wait-ratio NaN
>>>>>> network-io-rate NaN metadata-age 239.828
>>>>>> records-per-request-avg NaN record-retry-rate NaN
>>>>>> buffer-total-bytes 6.7108864E7 buffer-available-bytes
>>>>>> 6.7108864E7 topic.*TOPIC_NAME*.record-error-rate NaN
>>>>>> record-send-rate NaN select-rate NaN
>>>>>> node-2.outgoing-byte-rate NaN
>>>>>> topic.*TOPIC_NAME*.record-retry-rate NaN batch-size-max
>>>>>> -Infinity connection-creation-rate NaN
>>>>>> node-1.outgoing-byte-rate NaN topic.*TOPIC_NAME*.byte-rate
>>>>>> NaN waiting-threads 0.0 batch-size-avg NaN io-wait-ratio NaN
>>>>>> io-wait-time-ns-avg NaN io-ratio NaN
>>>>>> topic.TOPIC_NAME.record-send-rate NaN request-size-max
>>>>>> -Infinity record-size-avg NaN request-latency-max -Infinity
>>>>>> node-2.request-latency-max -Infinity record-queue-time-max
>>>>>> -Infinity node-2.response-rate NaN node-1.request-rate NaN
>>>>>> node-1.request-size-max -Infinity connection-count 3.0
>>>>>> incoming-byte-rate NaN compression-rate-avg NaN request-rate
>>>>>> NaN node-1.response-rate NaN node-2.request-latency-avg NaN
>>>>>> request-latency-avg NaN record-error-rate NaN
>>>>>> connection-close-rate NaN *node-2.*request-size-max
>>>>>> -Infinity topic.TOPIC_NAME.compression-rate NaN
>>>>>> node-2.incoming-byte-rate NaN node-1.request-size-avg NaN
>>>>>> io-time-ns-avg NaN outgoing-byte-rate NaN
>>>>>> *node-2*.request-rate NaN response-rate NaN
>>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Jun Rao <ju...@gmail.com>.
Actually, we do give you an error if an invalid topic is created through
CLI. When we add a create topic api, we can return the correct error code
too.
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic="te+dd"
--partitions 1 --replication-factor 1
Error while executing topic command topic name te+dd is illegal, contains a
character other than ASCII alphanumerics, '.', '_' and '-'
kafka.common.InvalidTopicException: topic name te+dd is illegal, contains a
character other than ASCII alphanumerics, '.', '_' and '-'
Thanks,
Jun
On Sun, Sep 14, 2014 at 1:46 AM, Michael G. Noll <mi...@michael-noll.com>
wrote:
> Wouldn't it be helpful to throw an error or a warning if the user
> tries to create a topic with an invalid name? Currently neither the
> API nor the CLI tools inform you that you are naming a topic in a way
> you shouldn't.
>
> And as Otis pointed out elsewhere in this thread this ties back into
> the JMX/MBean issues related the usage of "-"/dashes and
> "_"/underscores, which are unfortunately the only non-alphanumeric
> characters that are at your disposal to add reasonable "structure" to
> your Kafka topic names. (https://issues.apache.org/jira/browse/KAFKA-1481)
>
> Do you mind sharing your topic naming convention at LinkedIn?
>
> --Michael
>
>
>
>
>
> On 11.09.2014 00:24, Jun Rao wrote:
> > We actually don't allow "." in the topic name. Topic name can be
> > alpha-numeric plus "-" and "_".
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry
> > <mi...@gmail.com> wrote:
> >
> >> Thanks, I was using without JMX. I will go through doc. But how
> >> about Topic or Metric name Topic Name Convention or Metric Name
> >> Convention ? The dot notation with topic having a ".". Any
> >> future plan to enforce some stand rules.
> >>
> >> Thanks,
> >>
> >> Bhavesh
> >>
> >> On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com>
> >> wrote:
> >>
> >>> Hi Bhavesh,
> >>>
> >>> Each of those JMX attributes comes with documentation. If you
> >>> open up jconsole and attach to a jvm running the consumer you
> >>> should be able to read the descriptions for each attribute.
> >>>
> >>> -Jay
> >>>
> >>> On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> >>> <mi...@gmail.com> wrote:
> >>>> Kafka Team,
> >>>>
> >>>> Can you please let me know what each of following Metrics
> >>>> means ? Some
> >>> of
> >>>> them are obvious, but some are hard to understand. My Topic
> >>>> name is *TOPIC_NAME*.
> >>>>
> >>>>
> >>>>
> >>>> can we enforce a Topic Name Convention or Metric Name
> >>>> Convention.
> >>> Because
> >>>> in previous version of Kafka, we have similar issue of
> >>>> parsing Kafka Metrics name with host name issue (codahale
> >>>> lib) . I have topic name
> >>> with
> >>>> “.” So, it is hard to distinguish metric name and topic.
> >>>> Also, when guys get chance I would appreciate if you guys
> >>>> can explain metric description on wiki so community would
> >>>> know what to monitor. Please
> >> see
> >>>> below for full list of metrics from new producer.
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Bhavesh
> >>>>
> >>>>
> >>>> record-queue-time-avg NaN *node-1.*request-latency-max
> >>>> -Infinity record-size-max -Infinity
> >>>> *node-1.*incoming-byte-rate NaN request-size-avg NaN
> >>>> *node-1.*request-latency-avg NaN *node-2.*request-size-avg
> >>>> NaN requests-in-flight 0.0 bufferpool-wait-ratio NaN
> >>>> network-io-rate NaN metadata-age 239.828
> >>>> records-per-request-avg NaN record-retry-rate NaN
> >>>> buffer-total-bytes 6.7108864E7 buffer-available-bytes
> >>>> 6.7108864E7 topic.*TOPIC_NAME*.record-error-rate NaN
> >>>> record-send-rate NaN select-rate NaN
> >>>> node-2.outgoing-byte-rate NaN
> >>>> topic.*TOPIC_NAME*.record-retry-rate NaN batch-size-max
> >>>> -Infinity connection-creation-rate NaN
> >>>> node-1.outgoing-byte-rate NaN topic.*TOPIC_NAME*.byte-rate
> >>>> NaN waiting-threads 0.0 batch-size-avg NaN io-wait-ratio NaN
> >>>> io-wait-time-ns-avg NaN io-ratio NaN
> >>>> topic.TOPIC_NAME.record-send-rate NaN request-size-max
> >>>> -Infinity record-size-avg NaN request-latency-max -Infinity
> >>>> node-2.request-latency-max -Infinity record-queue-time-max
> >>>> -Infinity node-2.response-rate NaN node-1.request-rate NaN
> >>>> node-1.request-size-max -Infinity connection-count 3.0
> >>>> incoming-byte-rate NaN compression-rate-avg NaN request-rate
> >>>> NaN node-1.response-rate NaN node-2.request-latency-avg NaN
> >>>> request-latency-avg NaN record-error-rate NaN
> >>>> connection-close-rate NaN *node-2.*request-size-max
> >>>> -Infinity topic.TOPIC_NAME.compression-rate NaN
> >>>> node-2.incoming-byte-rate NaN node-1.request-size-avg NaN
> >>>> io-time-ns-avg NaN outgoing-byte-rate NaN
> >>>> *node-2*.request-rate NaN response-rate NaN
> >>>
> >>
> >
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by "Michael G. Noll" <mi...@michael-noll.com>.
Wouldn't it be helpful to throw an error or a warning if the user
tries to create a topic with an invalid name? Currently neither the
API nor the CLI tools inform you that you are naming a topic in a way
you shouldn't.
And as Otis pointed out elsewhere in this thread this ties back into
the JMX/MBean issues related the usage of "-"/dashes and
"_"/underscores, which are unfortunately the only non-alphanumeric
characters that are at your disposal to add reasonable "structure" to
your Kafka topic names. (https://issues.apache.org/jira/browse/KAFKA-1481)
Do you mind sharing your topic naming convention at LinkedIn?
--Michael
On 11.09.2014 00:24, Jun Rao wrote:
> We actually don't allow "." in the topic name. Topic name can be
> alpha-numeric plus "-" and "_".
>
> Thanks,
>
> Jun
>
> On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry
> <mi...@gmail.com> wrote:
>
>> Thanks, I was using without JMX. I will go through doc. But how
>> about Topic or Metric name Topic Name Convention or Metric Name
>> Convention ? The dot notation with topic having a ".". Any
>> future plan to enforce some stand rules.
>>
>> Thanks,
>>
>> Bhavesh
>>
>> On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com>
>> wrote:
>>
>>> Hi Bhavesh,
>>>
>>> Each of those JMX attributes comes with documentation. If you
>>> open up jconsole and attach to a jvm running the consumer you
>>> should be able to read the descriptions for each attribute.
>>>
>>> -Jay
>>>
>>> On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
>>> <mi...@gmail.com> wrote:
>>>> Kafka Team,
>>>>
>>>> Can you please let me know what each of following Metrics
>>>> means ? Some
>>> of
>>>> them are obvious, but some are hard to understand. My Topic
>>>> name is *TOPIC_NAME*.
>>>>
>>>>
>>>>
>>>> can we enforce a Topic Name Convention or Metric Name
>>>> Convention.
>>> Because
>>>> in previous version of Kafka, we have similar issue of
>>>> parsing Kafka Metrics name with host name issue (codahale
>>>> lib) . I have topic name
>>> with
>>>> “.” So, it is hard to distinguish metric name and topic.
>>>> Also, when guys get chance I would appreciate if you guys
>>>> can explain metric description on wiki so community would
>>>> know what to monitor. Please
>> see
>>>> below for full list of metrics from new producer.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Bhavesh
>>>>
>>>>
>>>> record-queue-time-avg NaN *node-1.*request-latency-max
>>>> -Infinity record-size-max -Infinity
>>>> *node-1.*incoming-byte-rate NaN request-size-avg NaN
>>>> *node-1.*request-latency-avg NaN *node-2.*request-size-avg
>>>> NaN requests-in-flight 0.0 bufferpool-wait-ratio NaN
>>>> network-io-rate NaN metadata-age 239.828
>>>> records-per-request-avg NaN record-retry-rate NaN
>>>> buffer-total-bytes 6.7108864E7 buffer-available-bytes
>>>> 6.7108864E7 topic.*TOPIC_NAME*.record-error-rate NaN
>>>> record-send-rate NaN select-rate NaN
>>>> node-2.outgoing-byte-rate NaN
>>>> topic.*TOPIC_NAME*.record-retry-rate NaN batch-size-max
>>>> -Infinity connection-creation-rate NaN
>>>> node-1.outgoing-byte-rate NaN topic.*TOPIC_NAME*.byte-rate
>>>> NaN waiting-threads 0.0 batch-size-avg NaN io-wait-ratio NaN
>>>> io-wait-time-ns-avg NaN io-ratio NaN
>>>> topic.TOPIC_NAME.record-send-rate NaN request-size-max
>>>> -Infinity record-size-avg NaN request-latency-max -Infinity
>>>> node-2.request-latency-max -Infinity record-queue-time-max
>>>> -Infinity node-2.response-rate NaN node-1.request-rate NaN
>>>> node-1.request-size-max -Infinity connection-count 3.0
>>>> incoming-byte-rate NaN compression-rate-avg NaN request-rate
>>>> NaN node-1.response-rate NaN node-2.request-latency-avg NaN
>>>> request-latency-avg NaN record-error-rate NaN
>>>> connection-close-rate NaN *node-2.*request-size-max
>>>> -Infinity topic.TOPIC_NAME.compression-rate NaN
>>>> node-2.incoming-byte-rate NaN node-1.request-size-avg NaN
>>>> io-time-ns-avg NaN outgoing-byte-rate NaN
>>>> *node-2*.request-rate NaN response-rate NaN
>>>
>>
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Jun Rao <ju...@gmail.com>.
We actually don't allow "." in the topic name. Topic name can be
alpha-numeric plus "-" and "_".
Thanks,
Jun
On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Thanks, I was using without JMX. I will go through doc. But how about
> Topic or Metric name Topic Name Convention or Metric Name Convention ? The
> dot notation with topic having a ".". Any future plan to enforce some
> stand rules.
>
> Thanks,
>
> Bhavesh
>
> On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Hi Bhavesh,
> >
> > Each of those JMX attributes comes with documentation. If you open up
> > jconsole and attach to a jvm running the consumer you should be able
> > to read the descriptions for each attribute.
> >
> > -Jay
> >
> > On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> > <mi...@gmail.com> wrote:
> > > Kafka Team,
> > >
> > > Can you please let me know what each of following Metrics means ? Some
> > of
> > > them are obvious, but some are hard to understand. My Topic name is
> > > *TOPIC_NAME*.
> > >
> > >
> > >
> > > can we enforce a Topic Name Convention or Metric Name Convention.
> > Because
> > > in previous version of Kafka, we have similar issue of parsing Kafka
> > > Metrics name with host name issue (codahale lib) . I have topic name
> > with
> > > “.” So, it is hard to distinguish metric name and topic. Also, when
> > > guys get chance I would appreciate if you guys can explain metric
> > > description on wiki so community would know what to monitor. Please
> see
> > > below for full list of metrics from new producer.
> > >
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> > >
> > > record-queue-time-avg NaN
> > > *node-1.*request-latency-max -Infinity
> > > record-size-max -Infinity
> > > *node-1.*incoming-byte-rate NaN
> > > request-size-avg NaN
> > > *node-1.*request-latency-avg NaN
> > > *node-2.*request-size-avg NaN
> > > requests-in-flight 0.0
> > > bufferpool-wait-ratio NaN
> > > network-io-rate NaN
> > > metadata-age 239.828
> > > records-per-request-avg NaN
> > > record-retry-rate NaN
> > > buffer-total-bytes 6.7108864E7
> > > buffer-available-bytes 6.7108864E7
> > > topic.*TOPIC_NAME*.record-error-rate NaN
> > > record-send-rate NaN
> > > select-rate NaN
> > > node-2.outgoing-byte-rate NaN
> > > topic.*TOPIC_NAME*.record-retry-rate NaN
> > > batch-size-max -Infinity
> > > connection-creation-rate NaN
> > > node-1.outgoing-byte-rate NaN
> > > topic.*TOPIC_NAME*.byte-rate NaN
> > > waiting-threads 0.0
> > > batch-size-avg NaN
> > > io-wait-ratio NaN
> > > io-wait-time-ns-avg NaN
> > > io-ratio NaN
> > > topic.TOPIC_NAME.record-send-rate NaN
> > > request-size-max -Infinity
> > > record-size-avg NaN
> > > request-latency-max -Infinity
> > > node-2.request-latency-max -Infinity
> > > record-queue-time-max -Infinity
> > > node-2.response-rate NaN
> > > node-1.request-rate NaN
> > > node-1.request-size-max -Infinity
> > > connection-count 3.0
> > > incoming-byte-rate NaN
> > > compression-rate-avg NaN
> > > request-rate NaN
> > > node-1.response-rate NaN
> > > node-2.request-latency-avg NaN
> > > request-latency-avg NaN
> > > record-error-rate NaN
> > > connection-close-rate NaN
> > > *node-2.*request-size-max -Infinity
> > > topic.TOPIC_NAME.compression-rate NaN
> > > node-2.incoming-byte-rate NaN
> > > node-1.request-size-avg NaN
> > > io-time-ns-avg NaN
> > > outgoing-byte-rate NaN
> > > *node-2*.request-rate NaN
> > > response-rate NaN
> >
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Jun Rao <ju...@gmail.com>.
We actually don't allow "." in the topic name. Topic name can be
alpha-numeric plus "-" and "_".
Thanks,
Jun
On Tue, Sep 9, 2014 at 6:29 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Thanks, I was using without JMX. I will go through doc. But how about
> Topic or Metric name Topic Name Convention or Metric Name Convention ? The
> dot notation with topic having a ".". Any future plan to enforce some
> stand rules.
>
> Thanks,
>
> Bhavesh
>
> On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Hi Bhavesh,
> >
> > Each of those JMX attributes comes with documentation. If you open up
> > jconsole and attach to a jvm running the consumer you should be able
> > to read the descriptions for each attribute.
> >
> > -Jay
> >
> > On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> > <mi...@gmail.com> wrote:
> > > Kafka Team,
> > >
> > > Can you please let me know what each of following Metrics means ? Some
> > of
> > > them are obvious, but some are hard to understand. My Topic name is
> > > *TOPIC_NAME*.
> > >
> > >
> > >
> > > can we enforce a Topic Name Convention or Metric Name Convention.
> > Because
> > > in previous version of Kafka, we have similar issue of parsing Kafka
> > > Metrics name with host name issue (codahale lib) . I have topic name
> > with
> > > “.” So, it is hard to distinguish metric name and topic. Also, when
> > > guys get chance I would appreciate if you guys can explain metric
> > > description on wiki so community would know what to monitor. Please
> see
> > > below for full list of metrics from new producer.
> > >
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> > >
> > > record-queue-time-avg NaN
> > > *node-1.*request-latency-max -Infinity
> > > record-size-max -Infinity
> > > *node-1.*incoming-byte-rate NaN
> > > request-size-avg NaN
> > > *node-1.*request-latency-avg NaN
> > > *node-2.*request-size-avg NaN
> > > requests-in-flight 0.0
> > > bufferpool-wait-ratio NaN
> > > network-io-rate NaN
> > > metadata-age 239.828
> > > records-per-request-avg NaN
> > > record-retry-rate NaN
> > > buffer-total-bytes 6.7108864E7
> > > buffer-available-bytes 6.7108864E7
> > > topic.*TOPIC_NAME*.record-error-rate NaN
> > > record-send-rate NaN
> > > select-rate NaN
> > > node-2.outgoing-byte-rate NaN
> > > topic.*TOPIC_NAME*.record-retry-rate NaN
> > > batch-size-max -Infinity
> > > connection-creation-rate NaN
> > > node-1.outgoing-byte-rate NaN
> > > topic.*TOPIC_NAME*.byte-rate NaN
> > > waiting-threads 0.0
> > > batch-size-avg NaN
> > > io-wait-ratio NaN
> > > io-wait-time-ns-avg NaN
> > > io-ratio NaN
> > > topic.TOPIC_NAME.record-send-rate NaN
> > > request-size-max -Infinity
> > > record-size-avg NaN
> > > request-latency-max -Infinity
> > > node-2.request-latency-max -Infinity
> > > record-queue-time-max -Infinity
> > > node-2.response-rate NaN
> > > node-1.request-rate NaN
> > > node-1.request-size-max -Infinity
> > > connection-count 3.0
> > > incoming-byte-rate NaN
> > > compression-rate-avg NaN
> > > request-rate NaN
> > > node-1.response-rate NaN
> > > node-2.request-latency-avg NaN
> > > request-latency-avg NaN
> > > record-error-rate NaN
> > > connection-close-rate NaN
> > > *node-2.*request-size-max -Infinity
> > > topic.TOPIC_NAME.compression-rate NaN
> > > node-2.incoming-byte-rate NaN
> > > node-1.request-size-avg NaN
> > > io-time-ns-avg NaN
> > > outgoing-byte-rate NaN
> > > *node-2*.request-rate NaN
> > > response-rate NaN
> >
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Bhavesh Mistry <mi...@gmail.com>.
Thanks, I was using without JMX. I will go through doc. But how about
Topic or Metric name Topic Name Convention or Metric Name Convention ? The
dot notation with topic having a ".". Any future plan to enforce some
stand rules.
Thanks,
Bhavesh
On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com> wrote:
> Hi Bhavesh,
>
> Each of those JMX attributes comes with documentation. If you open up
> jconsole and attach to a jvm running the consumer you should be able
> to read the descriptions for each attribute.
>
> -Jay
>
> On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> <mi...@gmail.com> wrote:
> > Kafka Team,
> >
> > Can you please let me know what each of following Metrics means ? Some
> of
> > them are obvious, but some are hard to understand. My Topic name is
> > *TOPIC_NAME*.
> >
> >
> >
> > can we enforce a Topic Name Convention or Metric Name Convention.
> Because
> > in previous version of Kafka, we have similar issue of parsing Kafka
> > Metrics name with host name issue (codahale lib) . I have topic name
> with
> > “.” So, it is hard to distinguish metric name and topic. Also, when
> > guys get chance I would appreciate if you guys can explain metric
> > description on wiki so community would know what to monitor. Please see
> > below for full list of metrics from new producer.
> >
> >
> > Thanks,
> >
> > Bhavesh
> >
> >
> > record-queue-time-avg NaN
> > *node-1.*request-latency-max -Infinity
> > record-size-max -Infinity
> > *node-1.*incoming-byte-rate NaN
> > request-size-avg NaN
> > *node-1.*request-latency-avg NaN
> > *node-2.*request-size-avg NaN
> > requests-in-flight 0.0
> > bufferpool-wait-ratio NaN
> > network-io-rate NaN
> > metadata-age 239.828
> > records-per-request-avg NaN
> > record-retry-rate NaN
> > buffer-total-bytes 6.7108864E7
> > buffer-available-bytes 6.7108864E7
> > topic.*TOPIC_NAME*.record-error-rate NaN
> > record-send-rate NaN
> > select-rate NaN
> > node-2.outgoing-byte-rate NaN
> > topic.*TOPIC_NAME*.record-retry-rate NaN
> > batch-size-max -Infinity
> > connection-creation-rate NaN
> > node-1.outgoing-byte-rate NaN
> > topic.*TOPIC_NAME*.byte-rate NaN
> > waiting-threads 0.0
> > batch-size-avg NaN
> > io-wait-ratio NaN
> > io-wait-time-ns-avg NaN
> > io-ratio NaN
> > topic.TOPIC_NAME.record-send-rate NaN
> > request-size-max -Infinity
> > record-size-avg NaN
> > request-latency-max -Infinity
> > node-2.request-latency-max -Infinity
> > record-queue-time-max -Infinity
> > node-2.response-rate NaN
> > node-1.request-rate NaN
> > node-1.request-size-max -Infinity
> > connection-count 3.0
> > incoming-byte-rate NaN
> > compression-rate-avg NaN
> > request-rate NaN
> > node-1.response-rate NaN
> > node-2.request-latency-avg NaN
> > request-latency-avg NaN
> > record-error-rate NaN
> > connection-close-rate NaN
> > *node-2.*request-size-max -Infinity
> > topic.TOPIC_NAME.compression-rate NaN
> > node-2.incoming-byte-rate NaN
> > node-1.request-size-avg NaN
> > io-time-ns-avg NaN
> > outgoing-byte-rate NaN
> > *node-2*.request-rate NaN
> > response-rate NaN
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Bhavesh Mistry <mi...@gmail.com>.
Thanks, I was using without JMX. I will go through doc. But how about
Topic or Metric name Topic Name Convention or Metric Name Convention ? The
dot notation with topic having a ".". Any future plan to enforce some
stand rules.
Thanks,
Bhavesh
On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps <ja...@gmail.com> wrote:
> Hi Bhavesh,
>
> Each of those JMX attributes comes with documentation. If you open up
> jconsole and attach to a jvm running the consumer you should be able
> to read the descriptions for each attribute.
>
> -Jay
>
> On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
> <mi...@gmail.com> wrote:
> > Kafka Team,
> >
> > Can you please let me know what each of following Metrics means ? Some
> of
> > them are obvious, but some are hard to understand. My Topic name is
> > *TOPIC_NAME*.
> >
> >
> >
> > can we enforce a Topic Name Convention or Metric Name Convention.
> Because
> > in previous version of Kafka, we have similar issue of parsing Kafka
> > Metrics name with host name issue (codahale lib) . I have topic name
> with
> > “.” So, it is hard to distinguish metric name and topic. Also, when
> > guys get chance I would appreciate if you guys can explain metric
> > description on wiki so community would know what to monitor. Please see
> > below for full list of metrics from new producer.
> >
> >
> > Thanks,
> >
> > Bhavesh
> >
> >
> > record-queue-time-avg NaN
> > *node-1.*request-latency-max -Infinity
> > record-size-max -Infinity
> > *node-1.*incoming-byte-rate NaN
> > request-size-avg NaN
> > *node-1.*request-latency-avg NaN
> > *node-2.*request-size-avg NaN
> > requests-in-flight 0.0
> > bufferpool-wait-ratio NaN
> > network-io-rate NaN
> > metadata-age 239.828
> > records-per-request-avg NaN
> > record-retry-rate NaN
> > buffer-total-bytes 6.7108864E7
> > buffer-available-bytes 6.7108864E7
> > topic.*TOPIC_NAME*.record-error-rate NaN
> > record-send-rate NaN
> > select-rate NaN
> > node-2.outgoing-byte-rate NaN
> > topic.*TOPIC_NAME*.record-retry-rate NaN
> > batch-size-max -Infinity
> > connection-creation-rate NaN
> > node-1.outgoing-byte-rate NaN
> > topic.*TOPIC_NAME*.byte-rate NaN
> > waiting-threads 0.0
> > batch-size-avg NaN
> > io-wait-ratio NaN
> > io-wait-time-ns-avg NaN
> > io-ratio NaN
> > topic.TOPIC_NAME.record-send-rate NaN
> > request-size-max -Infinity
> > record-size-avg NaN
> > request-latency-max -Infinity
> > node-2.request-latency-max -Infinity
> > record-queue-time-max -Infinity
> > node-2.response-rate NaN
> > node-1.request-rate NaN
> > node-1.request-size-max -Infinity
> > connection-count 3.0
> > incoming-byte-rate NaN
> > compression-rate-avg NaN
> > request-rate NaN
> > node-1.response-rate NaN
> > node-2.request-latency-avg NaN
> > request-latency-avg NaN
> > record-error-rate NaN
> > connection-close-rate NaN
> > *node-2.*request-size-max -Infinity
> > topic.TOPIC_NAME.compression-rate NaN
> > node-2.incoming-byte-rate NaN
> > node-1.request-size-avg NaN
> > io-time-ns-avg NaN
> > outgoing-byte-rate NaN
> > *node-2*.request-rate NaN
> > response-rate NaN
>
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Jay Kreps <ja...@gmail.com>.
Hi Bhavesh,
Each of those JMX attributes comes with documentation. If you open up
jconsole and attach to a jvm running the consumer you should be able
to read the descriptions for each attribute.
-Jay
On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
<mi...@gmail.com> wrote:
> Kafka Team,
>
> Can you please let me know what each of following Metrics means ? Some of
> them are obvious, but some are hard to understand. My Topic name is
> *TOPIC_NAME*.
>
>
>
> can we enforce a Topic Name Convention or Metric Name Convention. Because
> in previous version of Kafka, we have similar issue of parsing Kafka
> Metrics name with host name issue (codahale lib) . I have topic name with
> “.” So, it is hard to distinguish metric name and topic. Also, when
> guys get chance I would appreciate if you guys can explain metric
> description on wiki so community would know what to monitor. Please see
> below for full list of metrics from new producer.
>
>
> Thanks,
>
> Bhavesh
>
>
> record-queue-time-avg NaN
> *node-1.*request-latency-max -Infinity
> record-size-max -Infinity
> *node-1.*incoming-byte-rate NaN
> request-size-avg NaN
> *node-1.*request-latency-avg NaN
> *node-2.*request-size-avg NaN
> requests-in-flight 0.0
> bufferpool-wait-ratio NaN
> network-io-rate NaN
> metadata-age 239.828
> records-per-request-avg NaN
> record-retry-rate NaN
> buffer-total-bytes 6.7108864E7
> buffer-available-bytes 6.7108864E7
> topic.*TOPIC_NAME*.record-error-rate NaN
> record-send-rate NaN
> select-rate NaN
> node-2.outgoing-byte-rate NaN
> topic.*TOPIC_NAME*.record-retry-rate NaN
> batch-size-max -Infinity
> connection-creation-rate NaN
> node-1.outgoing-byte-rate NaN
> topic.*TOPIC_NAME*.byte-rate NaN
> waiting-threads 0.0
> batch-size-avg NaN
> io-wait-ratio NaN
> io-wait-time-ns-avg NaN
> io-ratio NaN
> topic.TOPIC_NAME.record-send-rate NaN
> request-size-max -Infinity
> record-size-avg NaN
> request-latency-max -Infinity
> node-2.request-latency-max -Infinity
> record-queue-time-max -Infinity
> node-2.response-rate NaN
> node-1.request-rate NaN
> node-1.request-size-max -Infinity
> connection-count 3.0
> incoming-byte-rate NaN
> compression-rate-avg NaN
> request-rate NaN
> node-1.response-rate NaN
> node-2.request-latency-avg NaN
> request-latency-avg NaN
> record-error-rate NaN
> connection-close-rate NaN
> *node-2.*request-size-max -Infinity
> topic.TOPIC_NAME.compression-rate NaN
> node-2.incoming-byte-rate NaN
> node-1.request-size-avg NaN
> io-time-ns-avg NaN
> outgoing-byte-rate NaN
> *node-2*.request-rate NaN
> response-rate NaN
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Jay Kreps <ja...@gmail.com>.
Hi Bhavesh,
Each of those JMX attributes comes with documentation. If you open up
jconsole and attach to a jvm running the consumer you should be able
to read the descriptions for each attribute.
-Jay
On Tue, Sep 9, 2014 at 2:07 PM, Bhavesh Mistry
<mi...@gmail.com> wrote:
> Kafka Team,
>
> Can you please let me know what each of following Metrics means ? Some of
> them are obvious, but some are hard to understand. My Topic name is
> *TOPIC_NAME*.
>
>
>
> can we enforce a Topic Name Convention or Metric Name Convention. Because
> in previous version of Kafka, we have similar issue of parsing Kafka
> Metrics name with host name issue (codahale lib) . I have topic name with
> “.” So, it is hard to distinguish metric name and topic. Also, when
> guys get chance I would appreciate if you guys can explain metric
> description on wiki so community would know what to monitor. Please see
> below for full list of metrics from new producer.
>
>
> Thanks,
>
> Bhavesh
>
>
> record-queue-time-avg NaN
> *node-1.*request-latency-max -Infinity
> record-size-max -Infinity
> *node-1.*incoming-byte-rate NaN
> request-size-avg NaN
> *node-1.*request-latency-avg NaN
> *node-2.*request-size-avg NaN
> requests-in-flight 0.0
> bufferpool-wait-ratio NaN
> network-io-rate NaN
> metadata-age 239.828
> records-per-request-avg NaN
> record-retry-rate NaN
> buffer-total-bytes 6.7108864E7
> buffer-available-bytes 6.7108864E7
> topic.*TOPIC_NAME*.record-error-rate NaN
> record-send-rate NaN
> select-rate NaN
> node-2.outgoing-byte-rate NaN
> topic.*TOPIC_NAME*.record-retry-rate NaN
> batch-size-max -Infinity
> connection-creation-rate NaN
> node-1.outgoing-byte-rate NaN
> topic.*TOPIC_NAME*.byte-rate NaN
> waiting-threads 0.0
> batch-size-avg NaN
> io-wait-ratio NaN
> io-wait-time-ns-avg NaN
> io-ratio NaN
> topic.TOPIC_NAME.record-send-rate NaN
> request-size-max -Infinity
> record-size-avg NaN
> request-latency-max -Infinity
> node-2.request-latency-max -Infinity
> record-queue-time-max -Infinity
> node-2.response-rate NaN
> node-1.request-rate NaN
> node-1.request-size-max -Infinity
> connection-count 3.0
> incoming-byte-rate NaN
> compression-rate-avg NaN
> request-rate NaN
> node-1.response-rate NaN
> node-2.request-latency-avg NaN
> request-latency-avg NaN
> record-error-rate NaN
> connection-close-rate NaN
> *node-2.*request-size-max -Infinity
> topic.TOPIC_NAME.compression-rate NaN
> node-2.incoming-byte-rate NaN
> node-1.request-size-avg NaN
> io-time-ns-avg NaN
> outgoing-byte-rate NaN
> *node-2*.request-rate NaN
> response-rate NaN
Re: Need Document and Explanation Of New Metrics Name in New Java
Producer on Kafka Trunk
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
> I have topic name with “.” So, it is hard to distinguish metric name and
topic
If your topics or hosts have dashes or underscores or your topic has dots
you will have trouble extracting things from JMX. We provided a patch to
change delimiter character in MBeans and are hoping it gets included in
0.8.2.
See https://issues.apache.org/jira/browse/KAFKA-1481
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Sep 9, 2014 at 5:07 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Kafka Team,
>
> Can you please let me know what each of following Metrics means ? Some of
> them are obvious, but some are hard to understand. My Topic name is
> *TOPIC_NAME*.
>
>
>
> can we enforce a Topic Name Convention or Metric Name Convention. Because
> in previous version of Kafka, we have similar issue of parsing Kafka
> Metrics name with host name issue (codahale lib) . I have topic name with
> “.” So, it is hard to distinguish metric name and topic. Also, when
> guys get chance I would appreciate if you guys can explain metric
> description on wiki so community would know what to monitor. Please see
> below for full list of metrics from new producer.
>
>
> Thanks,
>
> Bhavesh
>
>
> record-queue-time-avg NaN
> *node-1.*request-latency-max -Infinity
> record-size-max -Infinity
> *node-1.*incoming-byte-rate NaN
> request-size-avg NaN
> *node-1.*request-latency-avg NaN
> *node-2.*request-size-avg NaN
> requests-in-flight 0.0
> bufferpool-wait-ratio NaN
> network-io-rate NaN
> metadata-age 239.828
> records-per-request-avg NaN
> record-retry-rate NaN
> buffer-total-bytes 6.7108864E7
> buffer-available-bytes 6.7108864E7
> topic.*TOPIC_NAME*.record-error-rate NaN
> record-send-rate NaN
> select-rate NaN
> node-2.outgoing-byte-rate NaN
> topic.*TOPIC_NAME*.record-retry-rate NaN
> batch-size-max -Infinity
> connection-creation-rate NaN
> node-1.outgoing-byte-rate NaN
> topic.*TOPIC_NAME*.byte-rate NaN
> waiting-threads 0.0
> batch-size-avg NaN
> io-wait-ratio NaN
> io-wait-time-ns-avg NaN
> io-ratio NaN
> topic.TOPIC_NAME.record-send-rate NaN
> request-size-max -Infinity
> record-size-avg NaN
> request-latency-max -Infinity
> node-2.request-latency-max -Infinity
> record-queue-time-max -Infinity
> node-2.response-rate NaN
> node-1.request-rate NaN
> node-1.request-size-max -Infinity
> connection-count 3.0
> incoming-byte-rate NaN
> compression-rate-avg NaN
> request-rate NaN
> node-1.response-rate NaN
> node-2.request-latency-avg NaN
> request-latency-avg NaN
> record-error-rate NaN
> connection-close-rate NaN
> *node-2.*request-size-max -Infinity
> topic.TOPIC_NAME.compression-rate NaN
> node-2.incoming-byte-rate NaN
> node-1.request-size-avg NaN
> io-time-ns-avg NaN
> outgoing-byte-rate NaN
> *node-2*.request-rate NaN
> response-rate NaN
>