How do I instantiate a metrics reporter in Kafka Streams, with custom config?

[James Cheng <wu...@gmail.com>]

Hi, we have a KafkaStreams app. We specify a custom metric reporter by doing:

        Properties config = new Properties();
        config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker1:9092");
        config.put(StreamsConfig.METRIC_REPORTER_CLASSES_CONFIG, "com.mycompany.MetricReporter");
        config.put("custom-key-for-metric-reporter", "value");

Previously, our metric reporter would get passed the properties object upon instantiation, and would retrieve its custom config. It appears that in recent releases, that in order to apply my metric reporter to the consumer, that I have to specify my config as "consumer.custom-key-for-metric-reporter". And for the producer, I have to specify it as "producer.custom-key-for-metric-reporter". If I don't prefix it with "consumer." or "producer." , it appears it gets stripped out from the properties object that is passsed to my metric reporter when the consumer/producer gets initialized, and so my metric reporter can't get its config.

That means that if I have a metrics reporter and I want to collect producer and consumer metrics, as well as kafka-streams metrics, that I have to specify my custom config 3 times:
1) consumer.custom-key-for-metric-reporter
2) producer.custom-key-for-metric-reporter
3) custom-key-for-metric-reporter

Is that behavior as designed or is that a bug? What is the desired behavior for non-recognized keys in the properties object?

And actually, for the metrics.reporter class itself, am I supposed to specify it as:

metrics.reporter

or

metric.reporters
producer.metric.reporters
consumer.metric.reporters

Thanks,
-James

Re: How do I instantiate a metrics reporter in Kafka Streams, with custom config?

[Guozhang Wang <wa...@gmail.com>]

Hi James,

Thanks for bringing this up.

The original design is to let users apply the prefix ONLY if they want to
override for the specific client. For example, for `metric.reporters` or
`interceptor.classes` which are defined for both producer and consumer (and
moving forward we are adding admin client as well), without the prefix they
will be applied to all embedded clients; and if users want to have
different config value overrides for different clients, they can use the
prefix.

So is you define for `metrics.reporter` it should be applied to both
producer and consumer (is it not the case for you? Please confirm).

For custom config k-v pairs, I think we did not discuss about this
in-depth. But our currently implementation is, indeed, restricting to
producer / consumer client prop definition only, for example.

getClientPropsWithPrefix(CONSUMER_PREFIX, ConsumerConfig.configNames());

props.putAll(originalsWithPrefix(prefix));


so if you do not add the prefix to your custom k-v pairs, it will be
skipped. Thinking about it a bit, I think we should probably do what you
were inclining to, i.e. to define custom k-v pairs also globally unless
overridden by prefix.

Could you file a JIRA for it?


Guozhang



On Wed, Nov 1, 2017 at 10:18 AM, James Cheng <wu...@gmail.com> wrote:

> Hi, we have a KafkaStreams app. We specify a custom metric reporter by
> doing:
>
>         Properties config = new Properties();
>         config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
> "kafka-broker1:9092");
>         config.put(StreamsConfig.METRIC_REPORTER_CLASSES_CONFIG,
> "com.mycompany.MetricReporter");
>         config.put("custom-key-for-metric-reporter", "value");
>
> Previously, our metric reporter would get passed the properties object
> upon instantiation, and would retrieve its custom config. It appears that
> in recent releases, that in order to apply my metric reporter to the
> consumer, that I have to specify my config as "consumer.custom-key-for-metric-reporter".
> And for the producer, I have to specify it as "producer.custom-key-for-metric-reporter".
> If I don't prefix it with "consumer." or "producer." , it appears it gets
> stripped out from the properties object that is passsed to my metric
> reporter when the consumer/producer gets initialized, and so my metric
> reporter can't get its config.
>
> That means that if I have a metrics reporter and I want to collect
> producer and consumer metrics, as well as kafka-streams metrics, that I
> have to specify my custom config 3 times:
> 1) consumer.custom-key-for-metric-reporter
> 2) producer.custom-key-for-metric-reporter
> 3) custom-key-for-metric-reporter
>
> Is that behavior as designed or is that a bug? What is the desired
> behavior for non-recognized keys in the properties object?
>
> And actually, for the metrics.reporter class itself, am I supposed to
> specify it as:
>
> metrics.reporter
>
> or
>
> metric.reporters
> producer.metric.reporters
> consumer.metric.reporters
>
> Thanks,
> -James
>
>


-- 
-- Guozhang