You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Thomas Becker <Th...@xperi.com> on 2020/07/28 16:47:16 UTC
RecordTooLargeException with old (0.10.0.0) consumer
We have some legacy applications using an old (0.10.0.0) version of the consumer that are hitting RecordTooLargeExceptions with the following message:
org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {mytopic-0=13920987} whose size is larger than the fetch size 1048576 and hence cannot be ever returned. Increase the fetch size, or decrease the maximum message size the broker will allow.
We have not increased the maximum message size on either the broker nor topic level, and I'm quite confident no messages approaching that size are in the topic. Further, even if I increase the max.partition.fetch.bytes to a very large value such as Integer.MAX_VALUE, the error still occurs. I stumbled across https://issues.apache.org/jira/browse/KAFKA-4762 which seems to match what we're seeing, but our messages are not compressed. But sure enough, a test application using the 0.10.1.0 consumer is able to consume the topic with no issues. Unfortunately upgrading our legacy applications is difficult for other reasons. Any ideas what's happening here?
--
[https://dts-web-images.s3.amazonaws.com/Images/email+signatures/xperi_117.png]
Tommy Becker
Principal Engineer
Pronouns: he/him/his
O: 919.460.4747
E: thomas.becker@xperi.com
www.xperi.com
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of Xperi is authorized to conclude any binding agreement on behalf of Xperi by email. Binding agreements with Xperi may only be made by a signed written agreement.
Re: RecordTooLargeException with old (0.10.0.0) consumer
Posted by Thomas Becker <Th...@xperi.com>.
Again, we haven't changed the default message size, I believe this exception is a red herring.
On Tue, 2020-07-28 at 17:38 +0000, Manoj.Agrawal2@cognizant.com wrote:
[EXTERNAL EMAIL] Attention: This email was sent from outside Xperi. DO NOT CLICK any links or attachments unless you expected them.
________________________________
Hi ,
You also make to change at producer and consumer side as well
server.properties:
message.max.bytes=15728640
replica.fetch.max.bytes=15728640
max.request.size=15728640
fetch.message.max.bytes=15728640
and producer.properties:
max.request.size=15728640
consumer
max.partition.fetch.bytes
On 7/28/20, 9:51 AM, "Thomas Becker" <Th...@xperi.com>> wrote:
[External]
We have some legacy applications using an old (0.10.0.0) version of the consumer that are hitting RecordTooLargeExceptions with the following message:
org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {mytopic-0=13920987} whose size is larger than the fetch size 1048576 and hence cannot be ever returned. Increase the fetch size, or decrease the maximum message size the broker will allow.
We have not increased the maximum message size on either the broker nor topic level, and I'm quite confident no messages approaching that size are in the topic. Further, even if I increase the max.partition.fetch.bytes to a very large value such as Integer.MAX_VALUE, the error still occurs. I stumbled across https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-4762&data=02%7C01%7CThomas.Becker%40tivo.com%7Cfd376717d3824bbde22d08d8331d40ae%7Cd05b7c6912014c0db45d7f1dcc227e4d%7C1%7C0%7C637315548016939647&sdata=%2FBPbP0E2r7XkMihp%2Btnu19s6OTfV655VS9%2BTRQ%2F6il4%3D&reserved=0 which seems to match what we're seeing, but our messages are not compressed. But sure enough, a test application using the 0.10.1.0 consumer is able to consume the topic with no issues. Unfortunately upgrading our legacy applications is difficult for other reasons. Any ideas what's happening here?
--
[https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdts-web-images.s3.amazonaws.com%2FImages%2Femail%2Bsignatures%2Fxperi_117.png&data=02%7C01%7CThomas.Becker%40tivo.com%7Cfd376717d3824bbde22d08d8331d40ae%7Cd05b7c6912014c0db45d7f1dcc227e4d%7C1%7C0%7C637315548016939647&sdata=6gCzFkW2s9X%2FvRZTs7Zzh1FJzFLNZk46k1%2BvZUkugD4%3D&reserved=0]
Tommy Becker
Principal Engineer
Pronouns: he/him/his
O: 919.460.4747
E: thomas.becker@xperi.com<ma...@xperi.com>
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.xperi.com%2F&data=02%7C01%7CThomas.Becker%40tivo.com%7Cfd376717d3824bbde22d08d8331d40ae%7Cd05b7c6912014c0db45d7f1dcc227e4d%7C1%7C0%7C637315548016939647&sdata=aND7C18XgVspkuEh6aRURyljGG4Qo3I8K8Ji5Lge9Jo%3D&reserved=0
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of Xperi is authorized to conclude any binding agreement on behalf of Xperi by email. Binding agreements with Xperi may only be made by a signed written agreement.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
--
[cid:f5f2359b5a2deb6470a7af30f647ae6ee88ae657.camel@tivo.com] Tommy Becker
Principal Engineer
Personalized Content Discovery
O +1 919.460.4747
tivo.com<http://www.tivo.com/>
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of Xperi is authorized to conclude any binding agreement on behalf of Xperi by email. Binding agreements with Xperi may only be made by a signed written agreement.
Re: RecordTooLargeException with old (0.10.0.0) consumer
Posted by Ma...@cognizant.com.
Hi ,
You also make to change at producer and consumer side as well
server.properties:
message.max.bytes=15728640
replica.fetch.max.bytes=15728640
max.request.size=15728640
fetch.message.max.bytes=15728640
and producer.properties:
max.request.size=15728640
consumer
max.partition.fetch.bytes
On 7/28/20, 9:51 AM, "Thomas Becker" <Th...@xperi.com> wrote:
[External]
We have some legacy applications using an old (0.10.0.0) version of the consumer that are hitting RecordTooLargeExceptions with the following message:
org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {mytopic-0=13920987} whose size is larger than the fetch size 1048576 and hence cannot be ever returned. Increase the fetch size, or decrease the maximum message size the broker will allow.
We have not increased the maximum message size on either the broker nor topic level, and I'm quite confident no messages approaching that size are in the topic. Further, even if I increase the max.partition.fetch.bytes to a very large value such as Integer.MAX_VALUE, the error still occurs. I stumbled across https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-4762&data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7C378f39e5a4054728a5b908d83316403a%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637315519004235818&sdata=JgvvsAxIRZUrpUhPAITjZ%2Fn9W8dU4WNuw1tX9ru87lE%3D&reserved=0 which seems to match what we're seeing, but our messages are not compressed. But sure enough, a test application using the 0.10.1.0 consumer is able to consume the topic with no issues. Unfortunately upgrading our legacy applications is difficult for other reasons. Any ideas what's happening here?
--
[https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdts-web-images.s3.amazonaws.com%2FImages%2Femail%2Bsignatures%2Fxperi_117.png&data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7C378f39e5a4054728a5b908d83316403a%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637315519004235818&sdata=qTznv4AlWuvbEDERyT2nQwgYb9I%2BA9M9nZzTGQw4IUY%3D&reserved=0]
Tommy Becker
Principal Engineer
Pronouns: he/him/his
O: 919.460.4747
E: thomas.becker@xperi.com
https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.xperi.com%2F&data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7C378f39e5a4054728a5b908d83316403a%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637315519004235818&sdata=p6WBnjPNrNi7sImgmufbASzqAm082fitWefOzyY4CZM%3D&reserved=0
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of Xperi is authorized to conclude any binding agreement on behalf of Xperi by email. Binding agreements with Xperi may only be made by a signed written agreement.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.