Re: Kafka Connectors output to topic.

[Mich Talebzadeh <mi...@gmail.com>]

Hi Nick,

Have you taken this further?

For example, have you considered using Spark Structured Streaming as an
effective ETL tool here.

You want to do some ETL then, you can define your micro-batch to get
your topic ingested from Kafka upstream, write your own transformation in
your language of choice, Python, Scala, Javas etc and then feed data to
sink. It handles json data as well.


 HTH

   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Thu, 11 Mar 2021 at 18:27, Nick Siviglia <ni...@gmail.com> wrote:

> Hi Everyone,
>
> I'd like to create a Kafka connector that instead of acting like a source
> or sink will instead do some processing on the data and output to another
> kafka topic. Has anyone done this before? Does anyone see any potential
> drawbacks?
>
> Data is json format at around 150 string and number fields per object. And
> I'm planning on receiving about 2 million a day.
>
> Thanks for any help,
> Nick
>