[jira] [Commented] (KAFKA-3494) mbeans overwritten with identical clients on a single jvm

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271902#comment-15271902 ] 

ASF GitHub Bot commented on KAFKA-3494:
---------------------------------------

GitHub user onurkaraman opened a pull request:

    https://github.com/apache/kafka/pull/1323

    KAFKA-3494: add metric id to client mbeans

    KafkaConsumer and KafkaProducer mbeans currently only have a client-id granularity. Client-id represents a logical name for an application.
    
    Given that quotas encourage reuse of client-ids, it's not unexpected to have multiple clients share the same client-id within the same jvm. When a later client gets instantiated with the same client-id as an earlier client in the same jvm, JmxReporter unregisters the original client's mbeans that collide with the new client's mbeans.
    
    This commit makes client mbeans have a metric-id granularity to prevent mbean collision and the original client mbean unregistration.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/onurkaraman/kafka KAFKA-3494

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/1323.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1323
    
----
commit 40a0189e856ba42be396bd2694e5719e8e29631b
Author: Onur Karaman <ok...@linkedin.com>
Date:   2016-05-04T23:34:28Z

    add metric id to client mbeans
    
    KafkaConsumer and KafkaProducer mbeans currently only have a client-id granularity. Client-id represents a logical name for an application.
    
    Given that quotas encourage reuse of client-ids, it's not unexpected to have multiple clients share the same client-id within the same jvm. When a later client gets instantiated with the same client-id as an earlier client in the same jvm, JmxReporter unregisters the original client's mbeans that collide with the new client's mbeans.
    
    This commit makes client mbeans have a metric-id granularity to prevent mbean collision and the original client mbean unregistration.

----


> mbeans overwritten with identical clients on a single jvm
> ---------------------------------------------------------
>
>                 Key: KAFKA-3494
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3494
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Onur Karaman
>
> Quotas today are implemented on a (client-id, broker) granularity. I think one of the motivating factors for using a simple quota id was to allow for flexibility in the granularity of the quota enforcement. For instance, entire services can share the same id to get some form of (service, broker) granularity quotas. From my understanding, client-id was chosen as the quota id because it's a property that already exists on the clients and reusing it had relatively low impact.
> Continuing the above example, let's say a service spins up multiple KafkaConsumers in one jvm sharing the same client-id because they want those consumers to be quotad as a single entity. Sharing client-ids within a single jvm would cause problems in client metrics since the mbeans tags only go as granular as the client-id.
> An easy example is kafka-metrics count. Here's a sample code snippet:
> {code}
> package org.apache.kafka.clients.consumer;
> import java.util.Collections;
> import java.util.Properties;
> import org.apache.kafka.common.TopicPartition;
> public class KafkaConsumerMetrics {
>     public static void main(String[] args) throws InterruptedException {
>         Properties properties = new Properties();
>         properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
>         properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "test");
>         properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
>         properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
>         properties.setProperty(ConsumerConfig.CLIENT_ID_CONFIG, "testclientid");
>         KafkaConsumer<String, String> kc1 = new KafkaConsumer<>(properties);
>         KafkaConsumer<String, String> kc2 = new KafkaConsumer<>(properties);
>         kc1.assign(Collections.singletonList(new TopicPartition("t1", 0)));
>         while (true) {
>             kc1.poll(1000);
>             System.out.println("kc1 metrics: " + kc1.metrics().size());
>             System.out.println("kc2 metrics: " + kc2.metrics().size());
>             Thread.sleep(1000);
>         }
>     }
> }
> {code}
> jconsole shows one mbean kafka.consumer:type=kafka-metrics-count,client-id=testclientid consistently with value 40.
> but stdout shows:
> {code}
> kc1 metrics: 69
> kc2 metrics: 40
> {code}
> I think the two possible solutions are:
> 1. add finer granularity to the mbeans to distinguish between the clients
> 2. require client ids to be unique per jvm like KafkaStreams has done



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)