You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Indu V <in...@suntecgroup.com> on 2020/06/08 13:17:20 UTC
Finding consumer Lag
Hi Team,
I am facing an issue in finding the lag (unconsumed record count) from Kafka against a topic. I have another process that has to be run after all records are consumed.
Each topic has given a single partition with the replication factor as 1.
Lag is computed by taking the difference of published offset and consumed offset.
In a transactional scenario, the published offset which is the partition end offset includes control messages. In my understanding, this is internally added by Kafka to handle the transaction commit and rollback.
I got count mismatches in a transaction rollback scenario. For a single unconsumed record, it is showing lag as 3.
So is there any way I can exclude the control messages from the lag.
Regards,
Indu V
This electronic mail (including any attachment thereto) may be confidential and privileged and is intended only for the individual or entity named above. Any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. Accordingly, if you are not the intended recipient, please notify the sender by replying to this email immediately and delete this email (and any attachment thereto) from your computer system...Thank You.
Re: Finding consumer Lag
Posted by Indu V <in...@suntecgroup.com>.
We are inserting 50,000 records to Kafka from our application. Its a single thread and transactional.
Regards,
Indu V
________________________________
From: Manoj.Agrawal2@cognizant.com <Ma...@cognizant.com>
Sent: 12 August 2020 10:51
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: Re: Finding consumer Lag
[External Email]: DO NOT CLICK on links or open attachments unless you recognise the sender and know the content is safe.
Can you please share what action you are performing and how ?
On 8/11/20, 10:19 PM, "Indu V" <in...@suntecgroup.com> wrote:
[External]
Hi Team,
I am facing an issue in a clustered Kafka environment,
org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors.
at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:279) ~[kafka-clients-2.0.1.jar:?]
Brokers - 3
Replication factor - 3
ISR - 2
Kafka version - kafka_2.12-2.5.0
Please help me to solve this issue.
Regards,
Indu V
This electronic mail (including any attachment thereto) may be confidential and privileged and is intended only for the individual or entity named above. Any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. Accordingly, if you are not the intended recipient, please notify the sender by replying to this email immediately and delete this email (and any attachment thereto) from your computer system...Thank You.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
Re: Finding consumer Lag
Posted by Ma...@cognizant.com.
Can you please share what action you are performing and how ?
On 8/11/20, 10:19 PM, "Indu V" <in...@suntecgroup.com> wrote:
[External]
Hi Team,
I am facing an issue in a clustered Kafka environment,
org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors.
at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:279) ~[kafka-clients-2.0.1.jar:?]
Brokers - 3
Replication factor - 3
ISR - 2
Kafka version - kafka_2.12-2.5.0
Please help me to solve this issue.
Regards,
Indu V
This electronic mail (including any attachment thereto) may be confidential and privileged and is intended only for the individual or entity named above. Any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. Accordingly, if you are not the intended recipient, please notify the sender by replying to this email immediately and delete this email (and any attachment thereto) from your computer system...Thank You.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
Exception on Producer send
Posted by Indu V <in...@suntecgroup.com>.
Hi Team,
I am facing an issue in a clustered Kafka environment,
org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors.
at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:279) ~[kafka-clients-2.0.1.jar:?]
Brokers - 3
Replication factor - 3
ISR - 2
Kafka version - kafka_2.12-2.5.0
Please help me to solve this issue.
Regards,
Indu V
This electronic mail (including any attachment thereto) may be confidential and privileged and is intended only for the individual or entity named above. Any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. Accordingly, if you are not the intended recipient, please notify the sender by replying to this email immediately and delete this email (and any attachment thereto) from your computer system...Thank You.
Re: Finding consumer Lag
Posted by Indu V <in...@suntecgroup.com>.
Hi Team,
I am facing an issue in a clustered Kafka environment,
org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors.
at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:279) ~[kafka-clients-2.0.1.jar:?]
Brokers - 3
Replication factor - 3
ISR - 2
Kafka version - kafka_2.12-2.5.0
Please help me to solve this issue.
Regards,
Indu V
This electronic mail (including any attachment thereto) may be confidential and privileged and is intended only for the individual or entity named above. Any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. Accordingly, if you are not the intended recipient, please notify the sender by replying to this email immediately and delete this email (and any attachment thereto) from your computer system...Thank You.
Re: Finding consumer Lag
Posted by Kumar Jayanti <ku...@gmail.com>.
Hi,
Did you checkout the following command
kafka-consumer-groups.sh --zookeeper zkhost:2181/kafka-cluster --describe
--group groupname
This will return the LAG information and it is defined as :
“the difference between the consumer Current-Offset and the broker
Log-End-Offset for this topic partition”
On Mon, Jun 8, 2020 at 6:48 PM Indu V <in...@suntecgroup.com> wrote:
> Hi Team,
>
> I am facing an issue in finding the lag (unconsumed record count) from
> Kafka against a topic. I have another process that has to be run after all
> records are consumed.
>
> Each topic has given a single partition with the replication factor as 1.
>
> Lag is computed by taking the difference of published offset and consumed
> offset.
>
> In a transactional scenario, the published offset which is the partition
> end offset includes control messages. In my understanding, this is
> internally added by Kafka to handle the transaction commit and rollback.
>
> I got count mismatches in a transaction rollback scenario. For a single
> unconsumed record, it is showing lag as 3.
>
> So is there any way I can exclude the control messages from the lag.
>
> Regards,
> Indu V
>
> This electronic mail (including any attachment thereto) may be
> confidential and privileged and is intended only for the individual or
> entity named above. Any unauthorized use, printing, copying, disclosure or
> dissemination of this communication may be subject to legal restriction or
> sanction. Accordingly, if you are not the intended recipient, please notify
> the sender by replying to this email immediately and delete this email (and
> any attachment thereto) from your computer system...Thank You.
>
--
~~~~~~~~~~~~~~~~~~~~~~
https://kumarjayanti.github.io