[jira] [Resolved] (KAFKA-4747) add metrics for KafkaConsumer.poll

["Onur Karaman (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-4747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Onur Karaman resolved KAFKA-4747.
---------------------------------
    Resolution: Won't Fix

[~junrao] pointed out that the distinction between tim-in-poll and time-in-application can be effectively computed as 1 - (io-ratio) - (io-wait-ratio). If this value is close to 1, then time is mostly being spent on the application-side. Otherwise if this value is close to 0, then time is mostly being spent on the client-side.

Here's a simple experiment I ran to verify:
{code}
/**
 * Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE
 * file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file
 * to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the
 * License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
 * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
 * specific language governing permissions and limitations under the License.
 */
package org.apache.kafka.clients.consumer;

import org.apache.kafka.common.TopicPartition;

import java.util.Collections;
import java.util.Properties;

public class SlowKafkaConsumer {
    public static void main(String[] args) throws InterruptedException {
        long pollTimeout = Long.valueOf(args[0]);
        long sleepDuration = Long.valueOf(args[1]);
        Properties props = new Properties();
        props.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9090");
        props.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "onur");
        props.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArrayDeserializer");
        props.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArrayDeserializer");
        KafkaConsumer<byte[], byte[]> kafkaConsumer = new KafkaConsumer<>(props);
        kafkaConsumer.assign(Collections.singletonList(new TopicPartition("t", 0)));
        kafkaConsumer.seekToBeginning(Collections.singletonList(new TopicPartition("t", 0)));
        while (true) {
            kafkaConsumer.poll(pollTimeout);
            Thread.sleep(sleepDuration);
        }
    }
}
{code}

{code}
no data
=======
> ./bin/kafka-run-class.sh org.apache.kafka.clients.consumer.SlowKafkaConsumer 2000 0
io-ratio ~ 0
io-wait-ratio ~ 0.99

> ./bin/kafka-run-class.sh org.apache.kafka.clients.consumer.SlowKafkaConsumer 2000 10000
io-ratio ~ 0
io-wait-ratio ~ [0.1, 0.2]

> ./bin/kafka-run-class.sh org.apache.kafka.clients.consumer.SlowKafkaConsumer 2000 20000
io-ratio ~ 0
io-wait-ratio ~ [0.05, 0.12]

with data
=========
> ./bin/kafka-producer-perf-test.sh --producer-props bootstrap.servers=localhost:9090 --topic t --throughput -1 --num-records 100000000 --record-size 1000

> ./bin/kafka-run-class.sh org.apache.kafka.clients.consumer.SlowKafkaConsumer 2000 0
io-ratio ~ 0.06
io-wait-ratio ~ 0.8

> ./bin/kafka-run-class.sh org.apache.kafka.clients.consumer.SlowKafkaConsumer 2000 10000
io-ratio ~ 0
io-wait-ratio ~ [0.05, 0.1]

> ./bin/kafka-run-class.sh org.apache.kafka.clients.consumer.SlowKafkaConsumer 2000 20000
io-ratio ~ 0
io-wait-ratio ~ [0, 0.03]
{code}

> add metrics for KafkaConsumer.poll
> ----------------------------------
>
>                 Key: KAFKA-4747
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4747
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Onur Karaman
>            Assignee: Onur Karaman
>
> KafkaConsumer heavily depends on KafkaConsumer.poll yet we don't have metrics directly associated with it.
> We probably want to add two metrics:
> 1. time spent in KafkaConsumer.poll
> 2. time since last KafkaConsumer.poll (measured as now - endTimeOfLastPoll)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)