You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nick Siviglia <ni...@gmail.com> on 2021/03/11 17:55:16 UTC

Kafka Connectors output to topic.

Hi Everyone,

I'd like to create a Kafka connector that instead of acting like a source
or sink will instead do some processing on the data and output to another
kafka topic. Has anyone done this before? Does anyone see any potential
drawbacks?

Data is json format at around 150 string and number fields per object. And
I'm planning on receiving about 2 million a day.

Thanks for any help,
Nick

Re: Kafka Connectors output to topic.

Posted by Robin Moffatt <ro...@confluent.io.INVALID>.
For what it's worth, that sounds less like a Kafka Connector and more like
a Kafka Streams app.


-- 

Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff


On Thu, 11 Mar 2021 at 18:27, Nick Siviglia <ni...@gmail.com> wrote:

> Hi Everyone,
>
> I'd like to create a Kafka connector that instead of acting like a source
> or sink will instead do some processing on the data and output to another
> kafka topic. Has anyone done this before? Does anyone see any potential
> drawbacks?
>
> Data is json format at around 150 string and number fields per object. And
> I'm planning on receiving about 2 million a day.
>
> Thanks for any help,
> Nick
>

Re: Kafka Connectors output to topic.

Posted by Mich Talebzadeh <mi...@gmail.com>.
so you want to do ETL on the input data

Kafka source (topic 1)  -> Processing -> Another Kafka sink (topic2)



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*





*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Thu, 11 Mar 2021 at 18:27, Nick Siviglia <ni...@gmail.com> wrote:

> Hi Everyone,
>
> I'd like to create a Kafka connector that instead of acting like a source
> or sink will instead do some processing on the data and output to another
> kafka topic. Has anyone done this before? Does anyone see any potential
> drawbacks?
>
> Data is json format at around 150 string and number fields per object. And
> I'm planning on receiving about 2 million a day.
>
> Thanks for any help,
> Nick
>

Re: Kafka Connectors output to topic.

Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Nick,

Have you taken this further?

For example, have you considered using Spark Structured Streaming as an
effective ETL tool here.

You want to do some ETL then, you can define your micro-batch to get
your topic ingested from Kafka upstream, write your own transformation in
your language of choice, Python, Scala, Javas etc and then feed data to
sink. It handles json data as well.


 HTH

   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Thu, 11 Mar 2021 at 18:27, Nick Siviglia <ni...@gmail.com> wrote:

> Hi Everyone,
>
> I'd like to create a Kafka connector that instead of acting like a source
> or sink will instead do some processing on the data and output to another
> kafka topic. Has anyone done this before? Does anyone see any potential
> drawbacks?
>
> Data is json format at around 150 string and number fields per object. And
> I'm planning on receiving about 2 million a day.
>
> Thanks for any help,
> Nick
>

Re: Kafka Connectors output to topic.

Posted by Israel Ekpo <is...@gmail.com>.
Nick,

There will be drawbacks to this because you would be using the wrong tool
for the task. The Kafka ecosystem is very rich with different tools and you
should spend a little more time to understand what they do and are meant for

I believe the ecosystem component you are looking for is Kafka Streams. You
can also use KSQLDB for this if it is simple enough and you want to avoid
writing Java/Scala code

You should spend some time to understand each of the components of the
ecosystem and it will become clearer how they can be leveraged for stream
processing:

https://kafka.apache.org/documentation/#api

Kafka Brokers
Kafka Connect
Kafka Streams
Kafka Producers
Kafka Consumers
Kafka Admin Clients

Confluent Schema Registry
Confluent REST Proxy
Confluent KSQLDB

Take a look at all of the above and try to understand how they are inter
connected, dependent and used and it will bring better clarity

To directly answer your question, you do not need a connector to process
your events, you need to create a Kafka Streams app or depending on the use
case leverage KSQLDB for the task. The number of instances can scale up or
down as needed. The connect framework was not intended for what you are
attempting to use it for.

Please follow up with any additional questions you may have

I hope this was helpful


On Thu, Mar 11, 2021 at 1:27 PM Nick Siviglia <ni...@gmail.com> wrote:

> Hi Everyone,
>
> I'd like to create a Kafka connector that instead of acting like a source
> or sink will instead do some processing on the data and output to another
> kafka topic. Has anyone done this before? Does anyone see any potential
> drawbacks?
>
> Data is json format at around 150 string and number fields per object. And
> I'm planning on receiving about 2 million a day.
>
> Thanks for any help,
> Nick
>