How can I start writing a Copycat connector?

[James Cheng <jc...@tivo.com>]

Hi,

I'd like to write a Copycat connector. I saw on Github that the Copycat data APIs have been committed to trunk. How can I get started? I've never developed against trunk before. Do I git clone from trunk, or is there a pre-built set of jars from trunk that I can use?

I see that there is a sample connector in copycat/file. For development purposes, I could just create a peer to that. Obviously I would not check in my connector in that location, but would that be the best way to get started?

Thanks,
-James

Re: How can I start writing a Copycat connector?

[Ewen Cheslack-Postava <ew...@confluent.io>]

I'd highly recommend keeping your connector in a separate repo and
following the process Gwen suggested. That's how I'm working on connectors
(and convertalizers, as Gwen likes to call them).

Keep in mind that copycat-hdfs and copycat-jdbc haven't been updated to the
Kafka Copycat code yet, although that'll happen in the next day or two.
However, they are still a fairly good template for a project (assuming you
like Maven as your build tool).

If you want to make sure things are structured right, you might want to
start by taking the Maven structure of one of the projects, then dump a
renamed version of the file connectors in there. That'll give you something
you can get to a buildable & testable state that isn't very big. Then start
modifying it to work with the data source you *actually* want to work with.

-Ewen

On Mon, Aug 31, 2015 at 7:27 PM, Gwen Shapira <gw...@confluent.io> wrote:

> Hi James,
>
> This is so exciting :)
>
> While the APIs are still marked as @unstable, we don't have immediate plans
> to modify them - so now is a good time to write a connector. Just accept
> the possibility that few modifications may be needed before release (I'm
> trying to get Ewen to add mandatory connector versions to the API...)
>
> I typically clone trunk and then run ./gradlew install - which builds and
> places the jars in local mvn repository.
> Then I can use them as dependencies in other projects.
>
> IMO, starting with an existing example and modifying it is a good approach.
> You can look at few more examples here:
> https://github.com/confluentinc/copycat-hdfs
> https://github.com/confluentinc/copycat-jdbc
> (Those are still under active development... don't try them in production
> yet)
>
> Ewen will probably chime in with more advice :)
>
> Gwen
>
> On Mon, Aug 31, 2015 at 7:14 PM, James Cheng <jc...@tivo.com> wrote:
>
> > Hi,
> >
> > I'd like to write a Copycat connector. I saw on Github that the Copycat
> > data APIs have been committed to trunk. How can I get started? I've never
> > developed against trunk before. Do I git clone from trunk, or is there a
> > pre-built set of jars from trunk that I can use?
> >
> > I see that there is a sample connector in copycat/file. For development
> > purposes, I could just create a peer to that. Obviously I would not check
> > in my connector in that location, but would that be the best way to get
> > started?
> >
> > Thanks,
> > -James
> >
> >
>



-- 
Thanks,
Ewen

Re: How can I start writing a Copycat connector?

[Gwen Shapira <gw...@confluent.io>]

Hi James,

This is so exciting :)

While the APIs are still marked as @unstable, we don't have immediate plans
to modify them - so now is a good time to write a connector. Just accept
the possibility that few modifications may be needed before release (I'm
trying to get Ewen to add mandatory connector versions to the API...)

I typically clone trunk and then run ./gradlew install - which builds and
places the jars in local mvn repository.
Then I can use them as dependencies in other projects.

IMO, starting with an existing example and modifying it is a good approach.
You can look at few more examples here:
https://github.com/confluentinc/copycat-hdfs
https://github.com/confluentinc/copycat-jdbc
(Those are still under active development... don't try them in production
yet)

Ewen will probably chime in with more advice :)

Gwen

On Mon, Aug 31, 2015 at 7:14 PM, James Cheng <jc...@tivo.com> wrote:

> Hi,
>
> I'd like to write a Copycat connector. I saw on Github that the Copycat
> data APIs have been committed to trunk. How can I get started? I've never
> developed against trunk before. Do I git clone from trunk, or is there a
> pre-built set of jars from trunk that I can use?
>
> I see that there is a sample connector in copycat/file. For development
> purposes, I could just create a peer to that. Obviously I would not check
> in my connector in that location, but would that be the best way to get
> started?
>
> Thanks,
> -James
>
>