You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ali Nazemian <al...@gmail.com> on 2018/01/09 13:12:17 UTC
Best practice for publishing byte messages to Kafka
Hi All,
I was wondering whether there is any best practice/recommendation for
publishing byte messages to Kafka. Is there any specific Serializer that is
recommended for this matter?
Cheers,
Ali
Re: Best practice for publishing byte messages to Kafka
Posted by Xin Li <Xi...@trivago.com>.
Protobuf?
On 11.01.18, 01:33, "Ali Nazemian" <al...@gmail.com> wrote:
Oops, I was mistaken. I meant serialization of an object as a byte array
from the first place!
On Wed, Jan 10, 2018 at 3:20 PM, Thunder Stumpges <
thunder.stumpges@gmail.com> wrote:
> Byte Array is essentially "serialized" already isn't it? I mean the message
> itself is sent as a byte array, so the default byte array serializer is as
> "efficient" as it gets, as it's just sending your byte array through as the
> message... there's no serialization happening.
> -Thunder
>
> On Tue, Jan 9, 2018 at 8:17 PM Ali Nazemian <al...@gmail.com> wrote:
>
> > Thanks, Matt. Have you done any benchmarking to see how using different
> > Serializers may impact throughput/latency?
> >
> > Regards,
> > Ali
> >
> > On Wed, Jan 10, 2018 at 7:55 AM, Matt Farmer <ma...@frmr.me> wrote:
> >
> > > We use the default byte array serializer provided with Kafka and it
> works
> > > great for us.
> > >
> > > > On Jan 9, 2018, at 8:12 AM, Ali Nazemian <al...@gmail.com>
> > wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I was wondering whether there is any best practice/recommendation for
> > > > publishing byte messages to Kafka. Is there any specific Serializer
> > that
> > > is
> > > > recommended for this matter?
> > > >
> > > > Cheers,
> > > > Ali
> > >
> > >
> >
> >
> > --
> > A.Nazemian
> >
>
--
A.Nazemian
Re: Best practice for publishing byte messages to Kafka
Posted by Ali Nazemian <al...@gmail.com>.
Oops, I was mistaken. I meant serialization of an object as a byte array
from the first place!
On Wed, Jan 10, 2018 at 3:20 PM, Thunder Stumpges <
thunder.stumpges@gmail.com> wrote:
> Byte Array is essentially "serialized" already isn't it? I mean the message
> itself is sent as a byte array, so the default byte array serializer is as
> "efficient" as it gets, as it's just sending your byte array through as the
> message... there's no serialization happening.
> -Thunder
>
> On Tue, Jan 9, 2018 at 8:17 PM Ali Nazemian <al...@gmail.com> wrote:
>
> > Thanks, Matt. Have you done any benchmarking to see how using different
> > Serializers may impact throughput/latency?
> >
> > Regards,
> > Ali
> >
> > On Wed, Jan 10, 2018 at 7:55 AM, Matt Farmer <ma...@frmr.me> wrote:
> >
> > > We use the default byte array serializer provided with Kafka and it
> works
> > > great for us.
> > >
> > > > On Jan 9, 2018, at 8:12 AM, Ali Nazemian <al...@gmail.com>
> > wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I was wondering whether there is any best practice/recommendation for
> > > > publishing byte messages to Kafka. Is there any specific Serializer
> > that
> > > is
> > > > recommended for this matter?
> > > >
> > > > Cheers,
> > > > Ali
> > >
> > >
> >
> >
> > --
> > A.Nazemian
> >
>
--
A.Nazemian
Re: Best practice for publishing byte messages to Kafka
Posted by Thunder Stumpges <th...@gmail.com>.
Byte Array is essentially "serialized" already isn't it? I mean the message
itself is sent as a byte array, so the default byte array serializer is as
"efficient" as it gets, as it's just sending your byte array through as the
message... there's no serialization happening.
-Thunder
On Tue, Jan 9, 2018 at 8:17 PM Ali Nazemian <al...@gmail.com> wrote:
> Thanks, Matt. Have you done any benchmarking to see how using different
> Serializers may impact throughput/latency?
>
> Regards,
> Ali
>
> On Wed, Jan 10, 2018 at 7:55 AM, Matt Farmer <ma...@frmr.me> wrote:
>
> > We use the default byte array serializer provided with Kafka and it works
> > great for us.
> >
> > > On Jan 9, 2018, at 8:12 AM, Ali Nazemian <al...@gmail.com>
> wrote:
> > >
> > > Hi All,
> > >
> > > I was wondering whether there is any best practice/recommendation for
> > > publishing byte messages to Kafka. Is there any specific Serializer
> that
> > is
> > > recommended for this matter?
> > >
> > > Cheers,
> > > Ali
> >
> >
>
>
> --
> A.Nazemian
>
RE: Best practice for publishing byte messages to Kafka
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
Whatever you use I recommend some sort of wrapper since Kafka doesn't support any sort of metadata (like the version of the serialization format).
-Dave
-----Original Message-----
From: Matt Farmer [mailto:matt@frmr.me]
Sent: Thursday, January 11, 2018 8:56 AM
To: users@kafka.apache.org
Subject: Re: Best practice for publishing byte messages to Kafka
We use Thrift and Kryo, and we haven’t done any formal analysis recently. Performance numbers for those should be easy to find by googling around.
I can say that we push tens of thousands of messages per second at peak and serialization hasn’t been a cause of production lag in the entire year that I’ve been working on this application. That’s an imprecise analysis, but I would readily recommend either library in a case where performance is your only concern. (Though, Avro may prove to be a better choice than Thrift these days.)
> On Jan 9, 2018, at 11:17 PM, Ali Nazemian <al...@gmail.com> wrote:
>
> Thanks, Matt. Have you done any benchmarking to see how using
> different Serializers may impact throughput/latency?
>
> Regards,
> Ali
>
> On Wed, Jan 10, 2018 at 7:55 AM, Matt Farmer <ma...@frmr.me> wrote:
>
>> We use the default byte array serializer provided with Kafka and it
>> works great for us.
>>
>>> On Jan 9, 2018, at 8:12 AM, Ali Nazemian <al...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> I was wondering whether there is any best practice/recommendation
>>> for publishing byte messages to Kafka. Is there any specific
>>> Serializer that
>> is
>>> recommended for this matter?
>>>
>>> Cheers,
>>> Ali
>>
>>
>
>
> --
> A.Nazemian
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Best practice for publishing byte messages to Kafka
Posted by Matt Farmer <ma...@frmr.me>.
We use Thrift and Kryo, and we haven’t done any formal analysis recently. Performance numbers for those should be easy to find by googling around.
I can say that we push tens of thousands of messages per second at peak and serialization hasn’t been a cause of production lag in the entire year that I’ve been working on this application. That’s an imprecise analysis, but I would readily recommend either library in a case where performance is your only concern. (Though, Avro may prove to be a better choice than Thrift these days.)
> On Jan 9, 2018, at 11:17 PM, Ali Nazemian <al...@gmail.com> wrote:
>
> Thanks, Matt. Have you done any benchmarking to see how using different
> Serializers may impact throughput/latency?
>
> Regards,
> Ali
>
> On Wed, Jan 10, 2018 at 7:55 AM, Matt Farmer <ma...@frmr.me> wrote:
>
>> We use the default byte array serializer provided with Kafka and it works
>> great for us.
>>
>>> On Jan 9, 2018, at 8:12 AM, Ali Nazemian <al...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> I was wondering whether there is any best practice/recommendation for
>>> publishing byte messages to Kafka. Is there any specific Serializer that
>> is
>>> recommended for this matter?
>>>
>>> Cheers,
>>> Ali
>>
>>
>
>
> --
> A.Nazemian
Re: Best practice for publishing byte messages to Kafka
Posted by Ali Nazemian <al...@gmail.com>.
Thanks, Matt. Have you done any benchmarking to see how using different
Serializers may impact throughput/latency?
Regards,
Ali
On Wed, Jan 10, 2018 at 7:55 AM, Matt Farmer <ma...@frmr.me> wrote:
> We use the default byte array serializer provided with Kafka and it works
> great for us.
>
> > On Jan 9, 2018, at 8:12 AM, Ali Nazemian <al...@gmail.com> wrote:
> >
> > Hi All,
> >
> > I was wondering whether there is any best practice/recommendation for
> > publishing byte messages to Kafka. Is there any specific Serializer that
> is
> > recommended for this matter?
> >
> > Cheers,
> > Ali
>
>
--
A.Nazemian
Re: Best practice for publishing byte messages to Kafka
Posted by Matt Farmer <ma...@frmr.me>.
We use the default byte array serializer provided with Kafka and it works great for us.
> On Jan 9, 2018, at 8:12 AM, Ali Nazemian <al...@gmail.com> wrote:
>
> Hi All,
>
> I was wondering whether there is any best practice/recommendation for
> publishing byte messages to Kafka. Is there any specific Serializer that is
> recommended for this matter?
>
> Cheers,
> Ali
Re: Best practice for publishing byte messages to Kafka
Posted by Ron Arts <ro...@novantum.com>.
Benchmarks can be found here: https://eng.uber.com/trip-data-squeeze/
-- Ron
2018-01-09 14:12 GMT+01:00 Ali Nazemian <al...@gmail.com>:
> Hi All,
>
> I was wondering whether there is any best practice/recommendation for
> publishing byte messages to Kafka. Is there any specific Serializer that is
> recommended for this matter?
>
> Cheers,
> Ali
>
--
Novantum
ron.arts@novantum.com
Amsterdam