Kafka：Consumer get messages less than produced

["Xiao Yang (肖杨)" <xi...@sobey.com>]

Hello,
Our system is using kafka for message notification, and sometimes,  we monitered the consumer lag begin to increase.
We can see the messages are handled by consumer quickly in our logs,but slower than which are produced(Only less than 10 messages  per second).  
If we restart kafka service , all message in this topic are quickly handled.
How does this happened, or are there anything wrong with our usage? How to avoid this problem?


Kafka Service :  Cluster of 3 nodes, kafka_version 1.1.1
Cmd line: /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2G -Xms2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xloggc:/opt/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/opt/kafka/bin/../logs -Dlog4j.configuration=file:/opt/kafka/bin/../config/log4j.properties -cp /opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b32.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/commons-lang3-3.5.jar:/opt/kafka/bin/../libs/connect-api-1.1.1.jar:/opt/kafka/bin/../libs/connect-file-1.1.1.jar:/opt/kafka/bin/../libs/connect-json-1.1.1.jar:/opt/kafka/bin/../libs/connect-runtime-1.1.1.jar:/opt/kafka/bin/../libs/connect-transforms-1.1.1.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0-b32.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0-b32.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0-b32.jar:/opt/kafka/bin/../libs/jackson-annotations-2.9.6.jar:/opt/kafka/bin/../libs/jackson-core-2.9.6.jar:/opt/kafka/bin/../libs/jackson-databind-2.9.6.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.9.6.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.6.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.6.jar:/opt/kafka/bin/../libs/javassist-3.20.0-GA.jar:/opt/kafka/bin/../libs/javassist-3.21.0-GA.jar:/opt/kafka/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/bin/../libs/javax.inject-1.jar:/opt/kafka/bin/../libs/javax.inject-2.5.0-b32.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.25.1.jar:/opt/kafka/bin/../libs/jersey-common-2.25.1.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.25.1.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.25.1.jar:/opt/kafka/bin/../libs/jersey-guava-2.25.1.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.25.1.jar:/opt/kafka/bin/../libs/jersey-server-2.25.1.jar:/opt/kafka/bin/../libs/jetty-client-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-continuation-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-http-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-io-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-security-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-server-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-servlet-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-servlets-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-util-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/kafka-clients-1.1.1.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-1.1.1.jar:/opt/kafka/bin/../libs/kafka-streams-1.1.1.jar:/opt/kafka/bin/../libs/kafka-streams-examples-1.1.1.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-1.1.1.jar:/opt/kafka/bin/../libs/kafka-tools-1.1.1.jar:/opt/kafka/bin/../libs/kafka_2.11-1.1.1-sources.jar:/opt/kafka/bin/../libs/kafka_2.11-1.1.1-test-sources.jar:/opt/kafka/bin/../libs/kafka_2.11-1.1.1.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.4.1.jar:/opt/kafka/bin/../libs/maven-artifact-3.5.3.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.7.3.jar:/opt/kafka/bin/../libs/scala-library-2.11.12.jar:/opt/kafka/bin/../libs/scala-logging_2.11-3.8.0.jar:/opt/kafka/bin/../libs/scala-reflect-2.11.12.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.1.jar:/opt/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/bin/../libs/zkclient-0.10.jar:/opt/kafka/bin/../libs/zookeeper-3.4.10.jar -javaagent:/opt/kafka/jmx/jmx.jar=8094:/opt/kafka/jmx/kafka-agent.yaml kafka.Kafka /opt/kafka/config/server.properties



xiaoyang@sobey.com