You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nick Siviglia <ni...@gmail.com> on 2021/03/11 17:55:16 UTC
Kafka Connectors output to topic.
Hi Everyone,
I'd like to create a Kafka connector that instead of acting like a source
or sink will instead do some processing on the data and output to another
kafka topic. Has anyone done this before? Does anyone see any potential
drawbacks?
Data is json format at around 150 string and number fields per object. And
I'm planning on receiving about 2 million a day.
Thanks for any help,
Nick
Re: Kafka Connectors output to topic.
Posted by Robin Moffatt <ro...@confluent.io.INVALID>.
For what it's worth, that sounds less like a Kafka Connector and more like
a Kafka Streams app.
--
Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
On Thu, 11 Mar 2021 at 18:27, Nick Siviglia <ni...@gmail.com> wrote:
> Hi Everyone,
>
> I'd like to create a Kafka connector that instead of acting like a source
> or sink will instead do some processing on the data and output to another
> kafka topic. Has anyone done this before? Does anyone see any potential
> drawbacks?
>
> Data is json format at around 150 string and number fields per object. And
> I'm planning on receiving about 2 million a day.
>
> Thanks for any help,
> Nick
>
Re: Kafka Connectors output to topic.
Posted by Mich Talebzadeh <mi...@gmail.com>.
so you want to do ETL on the input data
Kafka source (topic 1) -> Processing -> Another Kafka sink (topic2)
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 11 Mar 2021 at 18:27, Nick Siviglia <ni...@gmail.com> wrote:
> Hi Everyone,
>
> I'd like to create a Kafka connector that instead of acting like a source
> or sink will instead do some processing on the data and output to another
> kafka topic. Has anyone done this before? Does anyone see any potential
> drawbacks?
>
> Data is json format at around 150 string and number fields per object. And
> I'm planning on receiving about 2 million a day.
>
> Thanks for any help,
> Nick
>
Re: Kafka Connectors output to topic.
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Nick,
Have you taken this further?
For example, have you considered using Spark Structured Streaming as an
effective ETL tool here.
You want to do some ETL then, you can define your micro-batch to get
your topic ingested from Kafka upstream, write your own transformation in
your language of choice, Python, Scala, Javas etc and then feed data to
sink. It handles json data as well.
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 11 Mar 2021 at 18:27, Nick Siviglia <ni...@gmail.com> wrote:
> Hi Everyone,
>
> I'd like to create a Kafka connector that instead of acting like a source
> or sink will instead do some processing on the data and output to another
> kafka topic. Has anyone done this before? Does anyone see any potential
> drawbacks?
>
> Data is json format at around 150 string and number fields per object. And
> I'm planning on receiving about 2 million a day.
>
> Thanks for any help,
> Nick
>
Re: Kafka Connectors output to topic.
Posted by Israel Ekpo <is...@gmail.com>.
Nick,
There will be drawbacks to this because you would be using the wrong tool
for the task. The Kafka ecosystem is very rich with different tools and you
should spend a little more time to understand what they do and are meant for
I believe the ecosystem component you are looking for is Kafka Streams. You
can also use KSQLDB for this if it is simple enough and you want to avoid
writing Java/Scala code
You should spend some time to understand each of the components of the
ecosystem and it will become clearer how they can be leveraged for stream
processing:
https://kafka.apache.org/documentation/#api
Kafka Brokers
Kafka Connect
Kafka Streams
Kafka Producers
Kafka Consumers
Kafka Admin Clients
Confluent Schema Registry
Confluent REST Proxy
Confluent KSQLDB
Take a look at all of the above and try to understand how they are inter
connected, dependent and used and it will bring better clarity
To directly answer your question, you do not need a connector to process
your events, you need to create a Kafka Streams app or depending on the use
case leverage KSQLDB for the task. The number of instances can scale up or
down as needed. The connect framework was not intended for what you are
attempting to use it for.
Please follow up with any additional questions you may have
I hope this was helpful
On Thu, Mar 11, 2021 at 1:27 PM Nick Siviglia <ni...@gmail.com> wrote:
> Hi Everyone,
>
> I'd like to create a Kafka connector that instead of acting like a source
> or sink will instead do some processing on the data and output to another
> kafka topic. Has anyone done this before? Does anyone see any potential
> drawbacks?
>
> Data is json format at around 150 string and number fields per object. And
> I'm planning on receiving about 2 million a day.
>
> Thanks for any help,
> Nick
>