You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Peter Penzov <pe...@gmail.com> on 2021/01/31 14:49:10 UTC
Send several Serialized Java objects under one Kafka Topic
Hello All,
I'm working on proof of concept for sending several Java Objects
under one Kafka Topic. More about the requirements:
https://stackoverflow.com/questions/65811681/design-kafka-consumers-and-producers-for-scalability
I managed to implement this working concept:
https://github.com/rcbandit111/skyobject_engine
In this code example I send several Java Objects under one Kafka Topic
and response is sent back into another Kafka Topic.
I would like to ask you what are the advantages and disadvantages in
this design of sending data between Consumer and Producer? I want to
use a serialized Java object, not JSON because I think it's more
optimized for performance.
Can you share your feedback, please?
Also is there some better way to implement this? Like for example
using Streaming API?
BR,
Peter
Re: Send several Serialized Java objects under one Kafka Topic
Posted by Peter Penzov <pe...@gmail.com>.
Thanks you all for the answer,
>Use Avro and a discriminated union / sum type to combine all the types.
Can you share a bit more about this approach, plase?
BR,
Pater
On Mon, Feb 1, 2021 at 12:54 AM Christopher Smith <cb...@gmail.com> wrote:
>
> Use Avro and a discriminated union / sum type to combine all the types.
>
> On Sun, Jan 31, 2021, 6:49 AM Peter Penzov <pe...@gmail.com> wrote:
>
> > Hello All,
> > I'm working on proof of concept for sending several Java Objects
> > under one Kafka Topic. More about the requirements:
> >
> > https://stackoverflow.com/questions/65811681/design-kafka-consumers-and-producers-for-scalability
> >
> > I managed to implement this working concept:
> > https://github.com/rcbandit111/skyobject_engine
> >
> > In this code example I send several Java Objects under one Kafka Topic
> > and response is sent back into another Kafka Topic.
> >
> > I would like to ask you what are the advantages and disadvantages in
> > this design of sending data between Consumer and Producer? I want to
> > use a serialized Java object, not JSON because I think it's more
> > optimized for performance.
> >
> > Can you share your feedback, please?
> >
> > Also is there some better way to implement this? Like for example
> > using Streaming API?
> >
> >
> > BR,
> > Peter
> >
Re: Send several Serialized Java objects under one Kafka Topic
Posted by Christopher Smith <cb...@gmail.com>.
Use Avro and a discriminated union / sum type to combine all the types.
On Sun, Jan 31, 2021, 6:49 AM Peter Penzov <pe...@gmail.com> wrote:
> Hello All,
> I'm working on proof of concept for sending several Java Objects
> under one Kafka Topic. More about the requirements:
>
> https://stackoverflow.com/questions/65811681/design-kafka-consumers-and-producers-for-scalability
>
> I managed to implement this working concept:
> https://github.com/rcbandit111/skyobject_engine
>
> In this code example I send several Java Objects under one Kafka Topic
> and response is sent back into another Kafka Topic.
>
> I would like to ask you what are the advantages and disadvantages in
> this design of sending data between Consumer and Producer? I want to
> use a serialized Java object, not JSON because I think it's more
> optimized for performance.
>
> Can you share your feedback, please?
>
> Also is there some better way to implement this? Like for example
> using Streaming API?
>
>
> BR,
> Peter
>
Re: Send several Serialized Java objects under one Kafka Topic
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Hi Peter,
The biggest issue with Java serialization has always been compatibility
between versions of the classes between producer and consumer, it can be
avoided via very careful management, but it's a lot more painful (IMO) than
other serialisation formats.
If you're looking to avoid JSON, I recommend Avro and/or Protobuf, as the
Schema Registry developed by Confluent and associated Kafka producer
serializers/ consumer deserializers explicitly support JSON, Avro and
Protobuf, and (once again, in my opinion) using the schema registry really
takes the pain out of schema version compatibility management.
Kind regards,
Liam Clarke-Hutchinson
On Mon, 1 Feb. 2021, 3:49 am Peter Penzov, <pe...@gmail.com> wrote:
> Hello All,
> I'm working on proof of concept for sending several Java Objects
> under one Kafka Topic. More about the requirements:
>
> https://stackoverflow.com/questions/65811681/design-kafka-consumers-and-producers-for-scalability
>
> I managed to implement this working concept:
> https://github.com/rcbandit111/skyobject_engine
>
> In this code example I send several Java Objects under one Kafka Topic
> and response is sent back into another Kafka Topic.
>
> I would like to ask you what are the advantages and disadvantages in
> this design of sending data between Consumer and Producer? I want to
> use a serialized Java object, not JSON because I think it's more
> optimized for performance.
>
> Can you share your feedback, please?
>
> Also is there some better way to implement this? Like for example
> using Streaming API?
>
>
> BR,
> Peter
>
Re: Send several Serialized Java objects under one Kafka Topic
Posted by Nicolas Carlot <ni...@chronopost.fr.INVALID>.
" I want to
use a serialized Java object, not JSON because I think it's more
optimized for performance."
=> Bad assumption.
https://www.robert-franz.com/2013/12/09/comparing-serialization-performance/
Avro and protobuf are oflently used, Json is easier
https://blog.softwaremill.com/the-best-serialization-strategy-for-event-sourcing-9321c299632b
Le dim. 31 janv. 2021 à 15:49, Peter Penzov <pe...@gmail.com> a
écrit :
> Hello All,
> I'm working on proof of concept for sending several Java Objects
> under one Kafka Topic. More about the requirements:
>
> https://stackoverflow.com/questions/65811681/design-kafka-consumers-and-producers-for-scalability
>
> I managed to implement this working concept:
> https://github.com/rcbandit111/skyobject_engine
>
> In this code example I send several Java Objects under one Kafka Topic
> and response is sent back into another Kafka Topic.
>
> I would like to ask you what are the advantages and disadvantages in
> this design of sending data between Consumer and Producer? I want to
> use a serialized Java object, not JSON because I think it's more
> optimized for performance.
>
> Can you share your feedback, please?
>
> Also is there some better way to implement this? Like for example
> using Streaming API?
>
>
> BR,
> Peter
>
--
*Nicolas Carlot*
| | nicolas.carlot@chronopost.fr
[image: Logo Chronopost]
| chronopost.fr <http://www.chronopost.fr/>
Suivez nous sur Facebook <https://fr-fr.facebook.com/chronopost> et Twitter
<https://twitter.com/chronopost>.
[image: DPD Group]