You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Sybrandy, Casey" <Ca...@Six3Systems.com> on 2012/02/15 23:37:46 UTC
Large message causing a MessageSizeTooLargeException
Hello,
I have a weird bug that I'm trying to figure out. I'm trying to send a 30+ MB message through Kafka, but it keeps giving me a MessageSizeTooLargeException. I did change the max.message.size property on the producer to be 50MB (50 * 1024 * 1024). Is there something I'm missing? The consumer is also set to accept messages up to the same size.
Kafka 0.6.0
Thanks.
Casey
RE: Large message causing a MessageSizeTooLargeException
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
Nevermind. I just realized when I started stripping my pom.xml to the bare necessities that I forgot to upgrade the Kafka client. Before, I only started the updated server. Now I'm getting other errors, but these appear to be with Zookeeper and potentially memory.
Thanks for your help.
Casey
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com]
Sent: Thursday, February 16, 2012 3:41 PM
To: kafka-users@incubator.apache.org
Subject: Re: Large message causing a MessageSizeTooLargeException
Casey,
This could be a bug. Do you mind filing a JIRA with a test case that
we can run to reproduce this problem ?
Thanks,
Neha
On Thu, Feb 16, 2012 at 12:24 PM, Sybrandy, Casey
<Ca...@six3systems.com> wrote:
> I'm not explicitly using an encoder. All of my messages are encoded using Avro and I send the resulting binary is passed to Kafka. That binary array is definitely less than the size of the buffer (About 37MB). Checking the size of that was one of the first things I did.
>
> This begs the question: is there a default encoder that's used? If so, could that be making the size larger than the buffer size?
>
> Casey
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Thursday, February 16, 2012 2:02 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> What encoder are you using? Is it possible that after encoding, the message
> size is larger than the threshold?
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 10:18 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
>> Yes, I do still see the exception. Is there a server setting that needs
>> to be updated?
>>
>> Has anyone else tried to push multi-megabyte files through Kafka?
>>
>> Casey
>> ________________________________________
>> From: Jun Rao [junrao@gmail.com]
>> Sent: Thursday, February 16, 2012 1:13 PM
>> To: kafka-users@incubator.apache.org
>> Subject: Re: Large message causing a MessageSizeTooLargeException
>>
>> There could be some transient failures in our unit tests especially on a
>> less powerful machine. Do you still see the exception?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey <
>> Casey.Sybrandy@six3systems.com> wrote:
>>
>> > Just tried it. No change. However, when I tried to create a release for
>> > it, it failed a number of unit tests. I resorted to just packaging it
>> and
>> > starting zookeeper and kafka in that directory and appeared to work fine,
>> > though I'm unsure if the failed tests have anything to do with it. I
>> don't
>> > suppose there's a pre-compiled release somewhere that I missed.
>> > ________________________________________
>> > From: Jun Rao [junrao@gmail.com]
>> > Sent: Thursday, February 16, 2012 11:35 AM
>> > To: kafka-users@incubator.apache.org
>> > Subject: Re: Large message causing a MessageSizeTooLargeException
>> >
>> > Could you try the 0.7 release, which is the first Apache release that we
>> > officially support?
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
>> > Casey.Sybrandy@six3systems.com> wrote:
>> >
>> > > No, I do not have compression enabled.
>> > >
>> > > Casey
>> > >
>> > > ________________________________________
>> > > From: Jun Rao [junrao@gmail.com]
>> > > Sent: Wednesday, February 15, 2012 6:31 PM
>> > > To: kafka-users@incubator.apache.org
>> > > Subject: Re: Large message causing a MessageSizeTooLargeException
>> > >
>> > > Did you enable compression? If so, try disable compression.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
>> > > Casey.Sybrandy@six3systems.com> wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > I have a weird bug that I'm trying to figure out. I'm trying to
>> send a
>> > > > 30+ MB message through Kafka, but it keeps giving me a
>> > > > MessageSizeTooLargeException. I did change the max.message.size
>> > property
>> > > > on the producer to be 50MB (50 * 1024 * 1024). Is there something
>> I'm
>> > > > missing? The consumer is also set to accept messages up to the same
>> > > size.
>> > > >
>> > > > Kafka 0.6.0
>> > > >
>> > > > Thanks.
>> > > >
>> > > > Casey
>> > >
>> >
>>
RE: Large message causing a MessageSizeTooLargeException
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
Neha,
No problem. I'll see what I can package together.
Casey
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com]
Sent: Thursday, February 16, 2012 3:41 PM
To: kafka-users@incubator.apache.org
Subject: Re: Large message causing a MessageSizeTooLargeException
Casey,
This could be a bug. Do you mind filing a JIRA with a test case that
we can run to reproduce this problem ?
Thanks,
Neha
On Thu, Feb 16, 2012 at 12:24 PM, Sybrandy, Casey
<Ca...@six3systems.com> wrote:
> I'm not explicitly using an encoder. All of my messages are encoded using Avro and I send the resulting binary is passed to Kafka. That binary array is definitely less than the size of the buffer (About 37MB). Checking the size of that was one of the first things I did.
>
> This begs the question: is there a default encoder that's used? If so, could that be making the size larger than the buffer size?
>
> Casey
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Thursday, February 16, 2012 2:02 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> What encoder are you using? Is it possible that after encoding, the message
> size is larger than the threshold?
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 10:18 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
>> Yes, I do still see the exception. Is there a server setting that needs
>> to be updated?
>>
>> Has anyone else tried to push multi-megabyte files through Kafka?
>>
>> Casey
>> ________________________________________
>> From: Jun Rao [junrao@gmail.com]
>> Sent: Thursday, February 16, 2012 1:13 PM
>> To: kafka-users@incubator.apache.org
>> Subject: Re: Large message causing a MessageSizeTooLargeException
>>
>> There could be some transient failures in our unit tests especially on a
>> less powerful machine. Do you still see the exception?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey <
>> Casey.Sybrandy@six3systems.com> wrote:
>>
>> > Just tried it. No change. However, when I tried to create a release for
>> > it, it failed a number of unit tests. I resorted to just packaging it
>> and
>> > starting zookeeper and kafka in that directory and appeared to work fine,
>> > though I'm unsure if the failed tests have anything to do with it. I
>> don't
>> > suppose there's a pre-compiled release somewhere that I missed.
>> > ________________________________________
>> > From: Jun Rao [junrao@gmail.com]
>> > Sent: Thursday, February 16, 2012 11:35 AM
>> > To: kafka-users@incubator.apache.org
>> > Subject: Re: Large message causing a MessageSizeTooLargeException
>> >
>> > Could you try the 0.7 release, which is the first Apache release that we
>> > officially support?
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
>> > Casey.Sybrandy@six3systems.com> wrote:
>> >
>> > > No, I do not have compression enabled.
>> > >
>> > > Casey
>> > >
>> > > ________________________________________
>> > > From: Jun Rao [junrao@gmail.com]
>> > > Sent: Wednesday, February 15, 2012 6:31 PM
>> > > To: kafka-users@incubator.apache.org
>> > > Subject: Re: Large message causing a MessageSizeTooLargeException
>> > >
>> > > Did you enable compression? If so, try disable compression.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
>> > > Casey.Sybrandy@six3systems.com> wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > I have a weird bug that I'm trying to figure out. I'm trying to
>> send a
>> > > > 30+ MB message through Kafka, but it keeps giving me a
>> > > > MessageSizeTooLargeException. I did change the max.message.size
>> > property
>> > > > on the producer to be 50MB (50 * 1024 * 1024). Is there something
>> I'm
>> > > > missing? The consumer is also set to accept messages up to the same
>> > > size.
>> > > >
>> > > > Kafka 0.6.0
>> > > >
>> > > > Thanks.
>> > > >
>> > > > Casey
>> > >
>> >
>>
Re: Large message causing a MessageSizeTooLargeException
Posted by Neha Narkhede <ne...@gmail.com>.
Casey,
This could be a bug. Do you mind filing a JIRA with a test case that
we can run to reproduce this problem ?
Thanks,
Neha
On Thu, Feb 16, 2012 at 12:24 PM, Sybrandy, Casey
<Ca...@six3systems.com> wrote:
> I'm not explicitly using an encoder. All of my messages are encoded using Avro and I send the resulting binary is passed to Kafka. That binary array is definitely less than the size of the buffer (About 37MB). Checking the size of that was one of the first things I did.
>
> This begs the question: is there a default encoder that's used? If so, could that be making the size larger than the buffer size?
>
> Casey
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Thursday, February 16, 2012 2:02 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> What encoder are you using? Is it possible that after encoding, the message
> size is larger than the threshold?
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 10:18 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
>> Yes, I do still see the exception. Is there a server setting that needs
>> to be updated?
>>
>> Has anyone else tried to push multi-megabyte files through Kafka?
>>
>> Casey
>> ________________________________________
>> From: Jun Rao [junrao@gmail.com]
>> Sent: Thursday, February 16, 2012 1:13 PM
>> To: kafka-users@incubator.apache.org
>> Subject: Re: Large message causing a MessageSizeTooLargeException
>>
>> There could be some transient failures in our unit tests especially on a
>> less powerful machine. Do you still see the exception?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey <
>> Casey.Sybrandy@six3systems.com> wrote:
>>
>> > Just tried it. No change. However, when I tried to create a release for
>> > it, it failed a number of unit tests. I resorted to just packaging it
>> and
>> > starting zookeeper and kafka in that directory and appeared to work fine,
>> > though I'm unsure if the failed tests have anything to do with it. I
>> don't
>> > suppose there's a pre-compiled release somewhere that I missed.
>> > ________________________________________
>> > From: Jun Rao [junrao@gmail.com]
>> > Sent: Thursday, February 16, 2012 11:35 AM
>> > To: kafka-users@incubator.apache.org
>> > Subject: Re: Large message causing a MessageSizeTooLargeException
>> >
>> > Could you try the 0.7 release, which is the first Apache release that we
>> > officially support?
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
>> > Casey.Sybrandy@six3systems.com> wrote:
>> >
>> > > No, I do not have compression enabled.
>> > >
>> > > Casey
>> > >
>> > > ________________________________________
>> > > From: Jun Rao [junrao@gmail.com]
>> > > Sent: Wednesday, February 15, 2012 6:31 PM
>> > > To: kafka-users@incubator.apache.org
>> > > Subject: Re: Large message causing a MessageSizeTooLargeException
>> > >
>> > > Did you enable compression? If so, try disable compression.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
>> > > Casey.Sybrandy@six3systems.com> wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > I have a weird bug that I'm trying to figure out. I'm trying to
>> send a
>> > > > 30+ MB message through Kafka, but it keeps giving me a
>> > > > MessageSizeTooLargeException. I did change the max.message.size
>> > property
>> > > > on the producer to be 50MB (50 * 1024 * 1024). Is there something
>> I'm
>> > > > missing? The consumer is also set to accept messages up to the same
>> > > size.
>> > > >
>> > > > Kafka 0.6.0
>> > > >
>> > > > Thanks.
>> > > >
>> > > > Casey
>> > >
>> >
>>
RE: Large message causing a MessageSizeTooLargeException
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
I'm not explicitly using an encoder. All of my messages are encoded using Avro and I send the resulting binary is passed to Kafka. That binary array is definitely less than the size of the buffer (About 37MB). Checking the size of that was one of the first things I did.
This begs the question: is there a default encoder that's used? If so, could that be making the size larger than the buffer size?
Casey
________________________________________
From: Jun Rao [junrao@gmail.com]
Sent: Thursday, February 16, 2012 2:02 PM
To: kafka-users@incubator.apache.org
Subject: Re: Large message causing a MessageSizeTooLargeException
What encoder are you using? Is it possible that after encoding, the message
size is larger than the threshold?
Thanks,
Jun
On Thu, Feb 16, 2012 at 10:18 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Yes, I do still see the exception. Is there a server setting that needs
> to be updated?
>
> Has anyone else tried to push multi-megabyte files through Kafka?
>
> Casey
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Thursday, February 16, 2012 1:13 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> There could be some transient failures in our unit tests especially on a
> less powerful machine. Do you still see the exception?
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > Just tried it. No change. However, when I tried to create a release for
> > it, it failed a number of unit tests. I resorted to just packaging it
> and
> > starting zookeeper and kafka in that directory and appeared to work fine,
> > though I'm unsure if the failed tests have anything to do with it. I
> don't
> > suppose there's a pre-compiled release somewhere that I missed.
> > ________________________________________
> > From: Jun Rao [junrao@gmail.com]
> > Sent: Thursday, February 16, 2012 11:35 AM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Large message causing a MessageSizeTooLargeException
> >
> > Could you try the 0.7 release, which is the first Apache release that we
> > officially support?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
> > Casey.Sybrandy@six3systems.com> wrote:
> >
> > > No, I do not have compression enabled.
> > >
> > > Casey
> > >
> > > ________________________________________
> > > From: Jun Rao [junrao@gmail.com]
> > > Sent: Wednesday, February 15, 2012 6:31 PM
> > > To: kafka-users@incubator.apache.org
> > > Subject: Re: Large message causing a MessageSizeTooLargeException
> > >
> > > Did you enable compression? If so, try disable compression.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
> > > Casey.Sybrandy@six3systems.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > I have a weird bug that I'm trying to figure out. I'm trying to
> send a
> > > > 30+ MB message through Kafka, but it keeps giving me a
> > > > MessageSizeTooLargeException. I did change the max.message.size
> > property
> > > > on the producer to be 50MB (50 * 1024 * 1024). Is there something
> I'm
> > > > missing? The consumer is also set to accept messages up to the same
> > > size.
> > > >
> > > > Kafka 0.6.0
> > > >
> > > > Thanks.
> > > >
> > > > Casey
> > >
> >
>
Re: Large message causing a MessageSizeTooLargeException
Posted by Jun Rao <ju...@gmail.com>.
What encoder are you using? Is it possible that after encoding, the message
size is larger than the threshold?
Thanks,
Jun
On Thu, Feb 16, 2012 at 10:18 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Yes, I do still see the exception. Is there a server setting that needs
> to be updated?
>
> Has anyone else tried to push multi-megabyte files through Kafka?
>
> Casey
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Thursday, February 16, 2012 1:13 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> There could be some transient failures in our unit tests especially on a
> less powerful machine. Do you still see the exception?
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > Just tried it. No change. However, when I tried to create a release for
> > it, it failed a number of unit tests. I resorted to just packaging it
> and
> > starting zookeeper and kafka in that directory and appeared to work fine,
> > though I'm unsure if the failed tests have anything to do with it. I
> don't
> > suppose there's a pre-compiled release somewhere that I missed.
> > ________________________________________
> > From: Jun Rao [junrao@gmail.com]
> > Sent: Thursday, February 16, 2012 11:35 AM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Large message causing a MessageSizeTooLargeException
> >
> > Could you try the 0.7 release, which is the first Apache release that we
> > officially support?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
> > Casey.Sybrandy@six3systems.com> wrote:
> >
> > > No, I do not have compression enabled.
> > >
> > > Casey
> > >
> > > ________________________________________
> > > From: Jun Rao [junrao@gmail.com]
> > > Sent: Wednesday, February 15, 2012 6:31 PM
> > > To: kafka-users@incubator.apache.org
> > > Subject: Re: Large message causing a MessageSizeTooLargeException
> > >
> > > Did you enable compression? If so, try disable compression.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
> > > Casey.Sybrandy@six3systems.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > I have a weird bug that I'm trying to figure out. I'm trying to
> send a
> > > > 30+ MB message through Kafka, but it keeps giving me a
> > > > MessageSizeTooLargeException. I did change the max.message.size
> > property
> > > > on the producer to be 50MB (50 * 1024 * 1024). Is there something
> I'm
> > > > missing? The consumer is also set to accept messages up to the same
> > > size.
> > > >
> > > > Kafka 0.6.0
> > > >
> > > > Thanks.
> > > >
> > > > Casey
> > >
> >
>
RE: Large message causing a MessageSizeTooLargeException
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
Yes, I do still see the exception. Is there a server setting that needs to be updated?
Has anyone else tried to push multi-megabyte files through Kafka?
Casey
________________________________________
From: Jun Rao [junrao@gmail.com]
Sent: Thursday, February 16, 2012 1:13 PM
To: kafka-users@incubator.apache.org
Subject: Re: Large message causing a MessageSizeTooLargeException
There could be some transient failures in our unit tests especially on a
less powerful machine. Do you still see the exception?
Thanks,
Jun
On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Just tried it. No change. However, when I tried to create a release for
> it, it failed a number of unit tests. I resorted to just packaging it and
> starting zookeeper and kafka in that directory and appeared to work fine,
> though I'm unsure if the failed tests have anything to do with it. I don't
> suppose there's a pre-compiled release somewhere that I missed.
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Thursday, February 16, 2012 11:35 AM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> Could you try the 0.7 release, which is the first Apache release that we
> officially support?
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > No, I do not have compression enabled.
> >
> > Casey
> >
> > ________________________________________
> > From: Jun Rao [junrao@gmail.com]
> > Sent: Wednesday, February 15, 2012 6:31 PM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Large message causing a MessageSizeTooLargeException
> >
> > Did you enable compression? If so, try disable compression.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
> > Casey.Sybrandy@six3systems.com> wrote:
> >
> > > Hello,
> > >
> > > I have a weird bug that I'm trying to figure out. I'm trying to send a
> > > 30+ MB message through Kafka, but it keeps giving me a
> > > MessageSizeTooLargeException. I did change the max.message.size
> property
> > > on the producer to be 50MB (50 * 1024 * 1024). Is there something I'm
> > > missing? The consumer is also set to accept messages up to the same
> > size.
> > >
> > > Kafka 0.6.0
> > >
> > > Thanks.
> > >
> > > Casey
> >
>
Re: Large message causing a MessageSizeTooLargeException
Posted by Jun Rao <ju...@gmail.com>.
There could be some transient failures in our unit tests especially on a
less powerful machine. Do you still see the exception?
Thanks,
Jun
On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Just tried it. No change. However, when I tried to create a release for
> it, it failed a number of unit tests. I resorted to just packaging it and
> starting zookeeper and kafka in that directory and appeared to work fine,
> though I'm unsure if the failed tests have anything to do with it. I don't
> suppose there's a pre-compiled release somewhere that I missed.
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Thursday, February 16, 2012 11:35 AM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> Could you try the 0.7 release, which is the first Apache release that we
> officially support?
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > No, I do not have compression enabled.
> >
> > Casey
> >
> > ________________________________________
> > From: Jun Rao [junrao@gmail.com]
> > Sent: Wednesday, February 15, 2012 6:31 PM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Large message causing a MessageSizeTooLargeException
> >
> > Did you enable compression? If so, try disable compression.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
> > Casey.Sybrandy@six3systems.com> wrote:
> >
> > > Hello,
> > >
> > > I have a weird bug that I'm trying to figure out. I'm trying to send a
> > > 30+ MB message through Kafka, but it keeps giving me a
> > > MessageSizeTooLargeException. I did change the max.message.size
> property
> > > on the producer to be 50MB (50 * 1024 * 1024). Is there something I'm
> > > missing? The consumer is also set to accept messages up to the same
> > size.
> > >
> > > Kafka 0.6.0
> > >
> > > Thanks.
> > >
> > > Casey
> >
>
RE: Large message causing a MessageSizeTooLargeException
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
Just tried it. No change. However, when I tried to create a release for it, it failed a number of unit tests. I resorted to just packaging it and starting zookeeper and kafka in that directory and appeared to work fine, though I'm unsure if the failed tests have anything to do with it. I don't suppose there's a pre-compiled release somewhere that I missed.
________________________________________
From: Jun Rao [junrao@gmail.com]
Sent: Thursday, February 16, 2012 11:35 AM
To: kafka-users@incubator.apache.org
Subject: Re: Large message causing a MessageSizeTooLargeException
Could you try the 0.7 release, which is the first Apache release that we
officially support?
Thanks,
Jun
On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> No, I do not have compression enabled.
>
> Casey
>
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Wednesday, February 15, 2012 6:31 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> Did you enable compression? If so, try disable compression.
>
> Thanks,
>
> Jun
>
> On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > Hello,
> >
> > I have a weird bug that I'm trying to figure out. I'm trying to send a
> > 30+ MB message through Kafka, but it keeps giving me a
> > MessageSizeTooLargeException. I did change the max.message.size property
> > on the producer to be 50MB (50 * 1024 * 1024). Is there something I'm
> > missing? The consumer is also set to accept messages up to the same
> size.
> >
> > Kafka 0.6.0
> >
> > Thanks.
> >
> > Casey
>
Re: Large message causing a MessageSizeTooLargeException
Posted by Jun Rao <ju...@gmail.com>.
Could you try the 0.7 release, which is the first Apache release that we
officially support?
Thanks,
Jun
On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> No, I do not have compression enabled.
>
> Casey
>
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Wednesday, February 15, 2012 6:31 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Large message causing a MessageSizeTooLargeException
>
> Did you enable compression? If so, try disable compression.
>
> Thanks,
>
> Jun
>
> On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > Hello,
> >
> > I have a weird bug that I'm trying to figure out. I'm trying to send a
> > 30+ MB message through Kafka, but it keeps giving me a
> > MessageSizeTooLargeException. I did change the max.message.size property
> > on the producer to be 50MB (50 * 1024 * 1024). Is there something I'm
> > missing? The consumer is also set to accept messages up to the same
> size.
> >
> > Kafka 0.6.0
> >
> > Thanks.
> >
> > Casey
>
RE: Large message causing a MessageSizeTooLargeException
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
No, I do not have compression enabled.
Casey
________________________________________
From: Jun Rao [junrao@gmail.com]
Sent: Wednesday, February 15, 2012 6:31 PM
To: kafka-users@incubator.apache.org
Subject: Re: Large message causing a MessageSizeTooLargeException
Did you enable compression? If so, try disable compression.
Thanks,
Jun
On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Hello,
>
> I have a weird bug that I'm trying to figure out. I'm trying to send a
> 30+ MB message through Kafka, but it keeps giving me a
> MessageSizeTooLargeException. I did change the max.message.size property
> on the producer to be 50MB (50 * 1024 * 1024). Is there something I'm
> missing? The consumer is also set to accept messages up to the same size.
>
> Kafka 0.6.0
>
> Thanks.
>
> Casey
Re: Large message causing a MessageSizeTooLargeException
Posted by Jun Rao <ju...@gmail.com>.
Did you enable compression? If so, try disable compression.
Thanks,
Jun
On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Hello,
>
> I have a weird bug that I'm trying to figure out. I'm trying to send a
> 30+ MB message through Kafka, but it keeps giving me a
> MessageSizeTooLargeException. I did change the max.message.size property
> on the producer to be 50MB (50 * 1024 * 1024). Is there something I'm
> missing? The consumer is also set to accept messages up to the same size.
>
> Kafka 0.6.0
>
> Thanks.
>
> Casey