You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2015/01/12 23:04:13 UTC
Kafka Cluster Monitoring and Documentation of Internals (JMX Metrics)
of Rejected Events
Hi Kafka Team,
I am trying to find out Kafka Internal and how a message can be corrupted
or lost at brokers side.
I have refer to following documentations for monitoring:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals
http://kafka.apache.org/documentation.html#monitoring
I am looking at following beans:
"kafka.server":type="BrokerTopicMetrics",name="test-FailedProduceRequestsPerSec"
"kafka.server":type="BrokerTopicMetrics",name="test-BytesRejectedPerSec"
I see following exception on Brokers side rejecting due to large request.
This is great but it does not show the source ip of prodcuer that caused
this issue ? Is there any way to log and capture this ?
*[2014-10-14 22:09:53,262] ERROR [KafkaApi-2] Error processing
ProducerRequest with correlation id 28795280 from client XXXXXon partition
[XXX,17]
(kafka.server.KafkaApis)kafka.common.MessageSizeTooLargeException: Message
size is 2924038 bytes which exceeds the maximum configured message size of
2097152. *
Can you this be reported as separate metric "MessageSizeTooLargeException
per topic" ?
Also, what is best way to find the CRC check error from the consumer side ?
How do you debug this ?
e.g log line:
*11 Dec 2014 07:22:33,387 ERROR [pool-15-thread-4] *
*kafka.message.InvalidMessageException: Message is corrupt (stored crc =
1834644195, computed crc = 2374999037)*
Also, is there any jira open to update with list all latest metrics and its
format and what it means ?
http://kafka.apache.org/documentation.html#monitoring. Please see attached
image for list of all metrics.
Version of Broker is 0.8.1.1.
Thanks,
Bhavesh
Re: Kafka Cluster Monitoring and Documentation of Internals (JMX
Metrics) of Rejected Events
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I think you could just email user@?
There was no attached image.
I think Jun committed something about this:
https://issues.apache.org/jira/browse/KAFKA-1481?focusedCommentId=14272057&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14272057
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Jan 12, 2015 at 5:04 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Kafka Team,
>
> I am trying to find out Kafka Internal and how a message can be corrupted
> or lost at brokers side.
>
> I have refer to following documentations for monitoring:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals
> http://kafka.apache.org/documentation.html#monitoring
>
> I am looking at following beans:
>
>
> "kafka.server":type="BrokerTopicMetrics",name="test-FailedProduceRequestsPerSec"
> "kafka.server":type="BrokerTopicMetrics",name="test-BytesRejectedPerSec"
>
>
> I see following exception on Brokers side rejecting due to large request.
> This is great but it does not show the source ip of prodcuer that caused
> this issue ? Is there any way to log and capture this ?
>
>
>
>
>
>
> *[2014-10-14 22:09:53,262] ERROR [KafkaApi-2] Error processing
> ProducerRequest with correlation id 28795280 from client XXXXXon partition
> [XXX,17]
> (kafka.server.KafkaApis)kafka.common.MessageSizeTooLargeException: Message
> size is 2924038 bytes which exceeds the maximum configured message size of
> 2097152. *
>
> Can you this be reported as separate metric "MessageSizeTooLargeException
> per topic" ?
>
> Also, what is best way to find the CRC check error from the consumer side
> ? How do you debug this ?
>
> e.g log line:
>
> *11 Dec 2014 07:22:33,387 ERROR [pool-15-thread-4] *
> *kafka.message.InvalidMessageException: Message is corrupt (stored crc =
> 1834644195, computed crc = 2374999037)*
>
> Also, is there any jira open to update with list all latest metrics and
> its format and what it means ?
> http://kafka.apache.org/documentation.html#monitoring. Please see
> attached image for list of all metrics.
>
> Version of Broker is 0.8.1.1.
>
> Thanks,
>
> Bhavesh
>
Re: Kafka Cluster Monitoring and Documentation of Internals (JMX
Metrics) of Rejected Events
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I think you could just email user@?
There was no attached image.
I think Jun committed something about this:
https://issues.apache.org/jira/browse/KAFKA-1481?focusedCommentId=14272057&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14272057
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Jan 12, 2015 at 5:04 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Kafka Team,
>
> I am trying to find out Kafka Internal and how a message can be corrupted
> or lost at brokers side.
>
> I have refer to following documentations for monitoring:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals
> http://kafka.apache.org/documentation.html#monitoring
>
> I am looking at following beans:
>
>
> "kafka.server":type="BrokerTopicMetrics",name="test-FailedProduceRequestsPerSec"
> "kafka.server":type="BrokerTopicMetrics",name="test-BytesRejectedPerSec"
>
>
> I see following exception on Brokers side rejecting due to large request.
> This is great but it does not show the source ip of prodcuer that caused
> this issue ? Is there any way to log and capture this ?
>
>
>
>
>
>
> *[2014-10-14 22:09:53,262] ERROR [KafkaApi-2] Error processing
> ProducerRequest with correlation id 28795280 from client XXXXXon partition
> [XXX,17]
> (kafka.server.KafkaApis)kafka.common.MessageSizeTooLargeException: Message
> size is 2924038 bytes which exceeds the maximum configured message size of
> 2097152. *
>
> Can you this be reported as separate metric "MessageSizeTooLargeException
> per topic" ?
>
> Also, what is best way to find the CRC check error from the consumer side
> ? How do you debug this ?
>
> e.g log line:
>
> *11 Dec 2014 07:22:33,387 ERROR [pool-15-thread-4] *
> *kafka.message.InvalidMessageException: Message is corrupt (stored crc =
> 1834644195, computed crc = 2374999037)*
>
> Also, is there any jira open to update with list all latest metrics and
> its format and what it means ?
> http://kafka.apache.org/documentation.html#monitoring. Please see
> attached image for list of all metrics.
>
> Version of Broker is 0.8.1.1.
>
> Thanks,
>
> Bhavesh
>
Re: Kafka Cluster Monitoring and Documentation of Internals (JMX
Metrics) of Rejected Events
Posted by Jun Rao <ju...@confluent.io>.
You can use the DumpLogSegments tool to see if a log segment is indeed
corrupted.
Thanks,
Jun
On Mon, Jan 12, 2015 at 2:04 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Kafka Team,
>
> I am trying to find out Kafka Internal and how a message can be corrupted
> or lost at brokers side.
>
> I have refer to following documentations for monitoring:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals
> http://kafka.apache.org/documentation.html#monitoring
>
> I am looking at following beans:
>
>
> "kafka.server":type="BrokerTopicMetrics",name="test-FailedProduceRequestsPerSec"
> "kafka.server":type="BrokerTopicMetrics",name="test-BytesRejectedPerSec"
>
>
> I see following exception on Brokers side rejecting due to large request.
> This is great but it does not show the source ip of prodcuer that caused
> this issue ? Is there any way to log and capture this ?
>
>
>
>
>
>
> *[2014-10-14 22:09:53,262] ERROR [KafkaApi-2] Error processing
> ProducerRequest with correlation id 28795280 from client XXXXXon partition
> [XXX,17]
> (kafka.server.KafkaApis)kafka.common.MessageSizeTooLargeException: Message
> size is 2924038 bytes which exceeds the maximum configured message size of
> 2097152. *
>
> Can you this be reported as separate metric "MessageSizeTooLargeException
> per topic" ?
>
> Also, what is best way to find the CRC check error from the consumer side
> ? How do you debug this ?
>
> e.g log line:
>
> *11 Dec 2014 07:22:33,387 ERROR [pool-15-thread-4] *
> *kafka.message.InvalidMessageException: Message is corrupt (stored crc =
> 1834644195, computed crc = 2374999037)*
>
> Also, is there any jira open to update with list all latest metrics and
> its format and what it means ?
> http://kafka.apache.org/documentation.html#monitoring. Please see
> attached image for list of all metrics.
>
> Version of Broker is 0.8.1.1.
>
> Thanks,
>
> Bhavesh
>