You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Eric Lachman <er...@spottradingllc.com> on 2016/01/27 20:41:26 UTC

Kakfa Connect Converter per Connector

Hi,

I am trying out Kafka connect and have a couple questions. We are directly publishing raw binary data to kafka from one of our apps and wanted to create a Kafka Connector Sink to move the raw data to something like Cassandra. Since this data is directly published to Kafka it doesn't have any of the Kafka Connect metadata such as Schema. So our Cassandra Sink Connector fails at parsing it. It seems we could write a custom converter to handle this raw data and fill out SchemaAndValue in very basic way to make it work. I'm not sure if this is the correct approach. If it is, it seems the only way to use the converter, at least in the stand alone mode, is to use the same converter for all of Connect, because it doesn't look like we could overwrite the converter config on a connector basis. So if we were to write new Sinks and Sources it would have to use our custom converter instead of the default.

Thanks,

Eric


________________________________

    Eric Lachman
     Software Developer I

     Spot Trading L.L.C
     440 South LaSalle St., Suite 2800
     Chicago, IL 60605
     Office: 312.362.4550
     Direct:
     Fax: 312.362.4551
     eric.lachman@spottradingllc.com
     www.spottradingllc.com<http://www.spottradingllc.com/>

________________________________

The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. Spot Trading, LLC

Re: Kakfa Connect Converter per Connector

Posted by eugene miretsky <eu...@gmail.com>.
Hi Eric,

We have a slightly different use case where we publish to Kafka using a
(modified) Connect Source and are using Spark Streaming to read the data
from Kafka and write to C* - it was really easy to write  simple code to
parse SchemaAndValue objects.

Setting up Spark Streaming is extremely easy (it has a built-in Kafka
consumer), and it provides all the fault tolerance, and distribution you
need. That said, it may be an overkill if all you need is to copy the data
over (we do a bunch of ETL in Spark Streaming).

A Connect Sink for Cassandra would be great. Let me know if you are indeed
planing to build and open source it, I'll be interested in contributing.

Cheers,

Eugene
On Jan 27, 2016 5:32 PM, "Gwen Shapira" <gw...@confluent.io> wrote:

> Hi Eric,
>
> 1. You are correct that the way to handle custom data formats in Kafka is
> to use a custom convertor.
> 2. You are also correct that we are currently assuming one converter per
> Connect instance / cluster that all connectors share (in the hope that each
> organization has one common data format...)
> If you need more data formats, the current work around is to run multiple
> KafkaConnect instances or clusters.
>
> I hope you have plans to open source your Cassandra connector! It will be
> very popular :)
> And if you decide to write your own Convertor, let us know if you have
> specific questions on how to proceed.
>
> Gwen
>
> On Wed, Jan 27, 2016 at 11:41 AM, Eric Lachman <
> eric.lachman@spottradingllc.com> wrote:
>
> > Hi,
> >
> > I am trying out Kafka connect and have a couple questions. We are
> directly
> > publishing raw binary data to kafka from one of our apps and wanted to
> > create a Kafka Connector Sink to move the raw data to something like
> > Cassandra. Since this data is directly published to Kafka it doesn't have
> > any of the Kafka Connect metadata such as Schema. So our Cassandra Sink
> > Connector fails at parsing it. It seems we could write a custom converter
> > to handle this raw data and fill out SchemaAndValue in very basic way to
> > make it work. I'm not sure if this is the correct approach. If it is, it
> > seems the only way to use the converter, at least in the stand alone
> mode,
> > is to use the same converter for all of Connect, because it doesn't look
> > like we could overwrite the converter config on a connector basis. So if
> we
> > were to write new Sinks and Sources it would have to use our custom
> > converter instead of the default.
> >
> > Thanks,
> >
> > Eric
> >
> >
> > ________________________________
> >
> >     Eric Lachman
> >      Software Developer I
> >
> >      Spot Trading L.L.C
> >      440 South LaSalle St., Suite 2800
> >      Chicago, IL 60605
> >      Office: 312.362.4550
> >      Direct:
> >      Fax: 312.362.4551
> >      eric.lachman@spottradingllc.com
> >      www.spottradingllc.com<http://www.spottradingllc.com/>
> >
> > ________________________________
> >
> > The information contained in this message may be privileged and
> > confidential and protected from disclosure. If the reader of this message
> > is not the intended recipient, or an employee or agent responsible for
> > delivering this message to the intended recipient, you are hereby
> notified
> > that any dissemination, distribution or copying of this communication is
> > strictly prohibited. If you have received this communication in error,
> > please notify us immediately by replying to the message and deleting it
> > from your computer. Thank you. Spot Trading, LLC
> >
> >
> >
>

Re: Kakfa Connect Converter per Connector

Posted by Gwen Shapira <gw...@confluent.io>.
Hi Eric,

1. You are correct that the way to handle custom data formats in Kafka is
to use a custom convertor.
2. You are also correct that we are currently assuming one converter per
Connect instance / cluster that all connectors share (in the hope that each
organization has one common data format...)
If you need more data formats, the current work around is to run multiple
KafkaConnect instances or clusters.

I hope you have plans to open source your Cassandra connector! It will be
very popular :)
And if you decide to write your own Convertor, let us know if you have
specific questions on how to proceed.

Gwen

On Wed, Jan 27, 2016 at 11:41 AM, Eric Lachman <
eric.lachman@spottradingllc.com> wrote:

> Hi,
>
> I am trying out Kafka connect and have a couple questions. We are directly
> publishing raw binary data to kafka from one of our apps and wanted to
> create a Kafka Connector Sink to move the raw data to something like
> Cassandra. Since this data is directly published to Kafka it doesn't have
> any of the Kafka Connect metadata such as Schema. So our Cassandra Sink
> Connector fails at parsing it. It seems we could write a custom converter
> to handle this raw data and fill out SchemaAndValue in very basic way to
> make it work. I'm not sure if this is the correct approach. If it is, it
> seems the only way to use the converter, at least in the stand alone mode,
> is to use the same converter for all of Connect, because it doesn't look
> like we could overwrite the converter config on a connector basis. So if we
> were to write new Sinks and Sources it would have to use our custom
> converter instead of the default.
>
> Thanks,
>
> Eric
>
>
> ________________________________
>
>     Eric Lachman
>      Software Developer I
>
>      Spot Trading L.L.C
>      440 South LaSalle St., Suite 2800
>      Chicago, IL 60605
>      Office: 312.362.4550
>      Direct:
>      Fax: 312.362.4551
>      eric.lachman@spottradingllc.com
>      www.spottradingllc.com<http://www.spottradingllc.com/>
>
> ________________________________
>
> The information contained in this message may be privileged and
> confidential and protected from disclosure. If the reader of this message
> is not the intended recipient, or an employee or agent responsible for
> delivering this message to the intended recipient, you are hereby notified
> that any dissemination, distribution or copying of this communication is
> strictly prohibited. If you have received this communication in error,
> please notify us immediately by replying to the message and deleting it
> from your computer. Thank you. Spot Trading, LLC
>
>
>