Copycat data API & serializers

[Ewen Cheslack-Postava <ew...@confluent.io>]

Hi all,

With the initial patch of of the Copycat APIs under review, I want to get
started on the data API. I wrote up some notes with a rough proposal for
what it should cover and a few requirements here:
https://cwiki.apache.org/confluence/display/KAFKA/Copycat+Data+API

Since they are very related, this also talks about serializers and includes
some sketches of how I think different serialization formats could be
implemented since we want Copycat to work with a variety of pluggable
serialization formats.

I think it's probably easy to bikeshed on the set of supported primitive
types or the semantics of optional fields & default values. But I think the
most important output from this discussion would be agreement that the
schemas described will be sufficiently rich for connectors, can handle
important cases like schema updates, but can still sufficiently abstract
away the underlying serialization libraries such that we aren't effectively
tied to one format despite providing a pluggable interface.

-- 
Thanks,
Ewen

Re: Copycat data API & serializers

[Ewen Cheslack-Postava <ew...@confluent.io>]

Bumping this thread so hopefully more people see it. There is also some
discussion in the corresponding JIRA:
https://issues.apache.org/jira/browse/KAFKA-2367

Any feedback is useful, even if just to say you personally don't have any
strong opinions on this topic.

Thanks,
-Ewen

On Fri, Jul 31, 2015 at 6:21 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:

> Hi all,
>
> With the initial patch of of the Copycat APIs under review, I want to get
> started on the data API. I wrote up some notes with a rough proposal for
> what it should cover and a few requirements here:
> https://cwiki.apache.org/confluence/display/KAFKA/Copycat+Data+API
>
> Since they are very related, this also talks about serializers and
> includes some sketches of how I think different serialization formats could
> be implemented since we want Copycat to work with a variety of pluggable
> serialization formats.
>
> I think it's probably easy to bikeshed on the set of supported primitive
> types or the semantics of optional fields & default values. But I think the
> most important output from this discussion would be agreement that the
> schemas described will be sufficiently rich for connectors, can handle
> important cases like schema updates, but can still sufficiently abstract
> away the underlying serialization libraries such that we aren't effectively
> tied to one format despite providing a pluggable interface.
>
> --
> Thanks,
> Ewen
>



-- 
Thanks,
Ewen