You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Sybrandy, Casey" <Ca...@Six3Systems.com> on 2013/05/08 14:57:25 UTC

Binary Data and Kafka

All,

Does the Kafka broker Base64 encode the messages?  We are sending binary data to the brokers and I looked at the logs to confirm that they data was being stored, however all of the data, with a few exceptions, looks to be Base64 encoded.  I didn't expect this, so I wanted to ask and confirm what I'm seeing.

If this is true, does this affect the size of the message when fetching?  In other words, if I send a 100K message, do I have to make sure I can fetch a 300K message since the message can now be 300K in size because of the encoding?

Casey Sybrandy MSWE
Six3Systems
Cyber and Enterprise Systems Group
www.six3systems.com
301-206-6000 (Office)
301-206-6020 (Fax)
11820 West Market Place
Suites N-P
Fulton, MD. 20759

RE: Binary Data and Kafka

Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
That's what I would have assumed.  And no, we're not using compression.

Thanks.

From: Jun Rao [mailto:junrao@gmail.com]
Sent: Wednesday, May 08, 2013 11:26 AM
To: users@kafka.apache.org
Cc: Sybrandy, Casey
Subject: Re: Binary Data and Kafka

No. Kafka broker stores the binary data as it is. The binary data may be compressed, if compression is enabled at the producer.

Thanks,

Jun

On Wed, May 8, 2013 at 5:57 AM, Sybrandy, Casey <Ca...@six3systems.com>> wrote:
All,

Does the Kafka broker Base64 encode the messages?  We are sending binary data to the brokers and I looked at the logs to confirm that they data was being stored, however all of the data, with a few exceptions, looks to be Base64 encoded.  I didn't expect this, so I wanted to ask and confirm what I'm seeing.

If this is true, does this affect the size of the message when fetching?  In other words, if I send a 100K message, do I have to make sure I can fetch a 300K message since the message can now be 300K in size because of the encoding?

Casey Sybrandy MSWE
Six3Systems
Cyber and Enterprise Systems Group
www.six3systems.com<http://www.six3systems.com>
301-206-6000<tel:301-206-6000> (Office)
301-206-6020<tel:301-206-6020> (Fax)
11820 West Market Place
Suites N-P
Fulton, MD. 20759

Re: Binary Data and Kafka

Posted by Jun Rao <ju...@gmail.com>.
No. Kafka broker stores the binary data as it is. The binary data may be
compressed, if compression is enabled at the producer.

Thanks,

Jun


On Wed, May 8, 2013 at 5:57 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:

> All,
>
> Does the Kafka broker Base64 encode the messages?  We are sending binary
> data to the brokers and I looked at the logs to confirm that they data was
> being stored, however all of the data, with a few exceptions, looks to be
> Base64 encoded.  I didn't expect this, so I wanted to ask and confirm what
> I'm seeing.
>
> If this is true, does this affect the size of the message when fetching?
>  In other words, if I send a 100K message, do I have to make sure I can
> fetch a 300K message since the message can now be 300K in size because of
> the encoding?
>
> Casey Sybrandy MSWE
> Six3Systems
> Cyber and Enterprise Systems Group
> www.six3systems.com
> 301-206-6000 (Office)
> 301-206-6020 (Fax)
> 11820 West Market Place
> Suites N-P
> Fulton, MD. 20759
>
>