how to work arounf Kafka message length?

[Vishal Kapoor <vi...@gmail.com>]

Hello,

We are in the process of building a general purpose messaging system, our
messages are in xml format to start with, and usually are longer than 1 mb,
what are the best practices around breaking messages into smaller units, we
may also need to support binary messages sometime in future, it it
advisable to increase the message length to say 128mb(biggest anticipated
message) when most of our messages might be within 5 mb?
we are open on using other technologies if they help solve the problem like
Avro etc...

Thanks,
Vishal

Re: how to work arounf Kafka message length?

[Guozhang Wang <wa...@gmail.com>]

Hi Vishal,

You can tune the message.max.bytes configs on the broker side to allow 1mb
messages along with compression:

https://kafka.apache.org/documentation.html#configuration

When you move on to 128mb then it would be a little hard to send the whole
message at once, using Avro to serialize the messages and then break the
byte arrays into pieces with offset header might work for you.

Guozhang




On Sat, Mar 22, 2014 at 6:45 PM, Dan Hoffman <ho...@gmail.com> wrote:

> Why not use the snappy/gzip compression?
>
> On Saturday, March 22, 2014, Vishal Kapoor <vi...@gmail.com>
> wrote:
>
> > Hello,
> >
> > We are in the process of building a general purpose messaging system, our
> > messages are in xml format to start with, and usually are longer than 1
> mb,
> > what are the best practices around breaking messages into smaller units,
> we
> > may also need to support binary messages sometime in future, it it
> > advisable to increase the message length to say 128mb(biggest anticipated
> > message) when most of our messages might be within 5 mb?
> > we are open on using other technologies if they help solve the problem
> like
> > Avro etc...
> >
> > Thanks,
> > Vishal
> >
>



-- 
-- Guozhang

Re: how to work arounf Kafka message length?

[Dan Hoffman <ho...@gmail.com>]

Why not use the snappy/gzip compression?

On Saturday, March 22, 2014, Vishal Kapoor <vi...@gmail.com>
wrote:

> Hello,
>
> We are in the process of building a general purpose messaging system, our
> messages are in xml format to start with, and usually are longer than 1 mb,
> what are the best practices around breaking messages into smaller units, we
> may also need to support binary messages sometime in future, it it
> advisable to increase the message length to say 128mb(biggest anticipated
> message) when most of our messages might be within 5 mb?
> we are open on using other technologies if they help solve the problem like
> Avro etc...
>
> Thanks,
> Vishal
>