gRPC with Kafka

["Glassman, Matthew S" <gl...@hsrc.unc.edu>]

I am involved with a course whose project has introduced REST, Kafka and gRPC on its topic of message passing.  However, in the project phase of this section, we are tasked with passing gRPC messages through Kafka to handle large amounts of location data.  Has anyone used these two items together?  The course teaches them separately and uses the kafka console scripts and so its a bit confusing trying to put the two together.  Should I utilize Kafka's python implementation?


Matthew Glassman

Re: gRPC with Kafka

[Israel Ekpo <is...@gmail.com>]

Hi Matthew,

If you can give more details about the course objectives that would help me
understand where your questions are coming from but I think they have to do
with the mixup between message formats and communication mechanisms.

I think you are mixing up the communication mechanisms with the message
formats.

REST and gRPC are communication mechanisms while JSON and Protocol Buffers
are message encoding formats.

Kafka has its own binary protocol that clients use when interacting with
the brokers but the message can be encoded in protocol buffer, JSON, Avro,
CSV or any binary format of your preference.

So when you say gRPC message, I assume you are referring to messages from
the producers encoded in protocol buffer format.

The Kafka connect framework and certain Kafka clients (producers,
consumers, streams) support using protocol buffers alongside the Confluent
Schema Registry) so you can use it in your apps in language this is
supported.

The console scripts are typically used when you want to demonstrate
client/broker interactions without using code so I can see how this could
confuse you especially for a binary message format like protocol buffer.

I think you should try to learn how to interact with the console
producers/consumers using simple data types like integers and strings and
then when you have mastered the concept you can move on to using python
code or Java code to interact with the brokers using protocol buffer, Avro
or any other complex message format.

I hope that helps.

Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://izzyacademy.com/



On Sat, Oct 30, 2021 at 9:37 AM Glassman, Matthew S <gl...@hsrc.unc.edu>
wrote:

> I am involved with a course whose project has introduced REST, Kafka and
> gRPC on its topic of message passing.  However, in the project phase of
> this section, we are tasked with passing gRPC messages through Kafka to
> handle large amounts of location data.  Has anyone used these two items
> together?  The course teaches them separately and uses the kafka console
> scripts and so its a bit confusing trying to put the two together.  Should
> I utilize Kafka's python implementation?
>
>
> Matthew Glassman
>
>
>
>