IO Authoring Guide - first draft

[Stephen Sisk <si...@google.com.INVALID>]

Here's the doc I've been working on:
https://docs.google.com/document/d/1nGGP2sLb5fLamB_dnkHVHC8BVjDD_SE46mQPIPkK5cQ/edit?usp=sharing


The general purpose of the doc is to share high level design thoughts and
process for authoring an IO transform. Topics include:
* List of example IOs that people can refer to, and for what reasons
* Steps for implementers
* Read transform design
* Write transform design (mostly a TODO for now)


Specifics of implementations, notably including the two items that started
recent discussions of the IO Authoring doc (the service interface and the
fake/mock pattern), are probably best covered by the transform style guide
that Eugene's working on and the "testing IO transforms" doc[1].

This is definitely a first draft. There is a lot that could be written on
the topic, and I have struggled a bit with how much detail to put in, but
since the jira issue has been open a while, this is a simple set of things
that I thought wouldn't be too controversial, and that would be forward
compatible with SplittableDoFn coming (thus, spending too much time
discussing sources isn't useful.)

I look forward to contributions from folks like Eugene, JB, and Dan who
have a lot more experience in this area. I'd like to get something up on
the website sooner rather than later, so if folks want to add large
chunks/sections, I might push us towards getting up a smaller, correct
version in a first PR, and folks can add more sections/design
patterns/etc... in later PRs.

Thanks!
Stephen

[1] Testing IO transforms -
https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing

Re: IO Authoring Guide - first draft

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Stephen,

Great work ! I will take a look during the week end and will provide 
feedbacks.

Thanks !
Regards
JB

On 01/28/2017 02:29 AM, Stephen Sisk wrote:
> Here's the doc I've been working on:
> https://docs.google.com/document/d/1nGGP2sLb5fLamB_dnkHVHC8BVjDD_SE46mQPIPkK5cQ/edit?usp=sharing
>
>
> The general purpose of the doc is to share high level design thoughts and
> process for authoring an IO transform. Topics include:
> * List of example IOs that people can refer to, and for what reasons
> * Steps for implementers
> * Read transform design
> * Write transform design (mostly a TODO for now)
>
>
> Specifics of implementations, notably including the two items that started
> recent discussions of the IO Authoring doc (the service interface and the
> fake/mock pattern), are probably best covered by the transform style guide
> that Eugene's working on and the "testing IO transforms" doc[1].
>
> This is definitely a first draft. There is a lot that could be written on
> the topic, and I have struggled a bit with how much detail to put in, but
> since the jira issue has been open a while, this is a simple set of things
> that I thought wouldn't be too controversial, and that would be forward
> compatible with SplittableDoFn coming (thus, spending too much time
> discussing sources isn't useful.)
>
> I look forward to contributions from folks like Eugene, JB, and Dan who
> have a lot more experience in this area. I'd like to get something up on
> the website sooner rather than later, so if folks want to add large
> chunks/sections, I might push us towards getting up a smaller, correct
> version in a first PR, and folks can add more sections/design
> patterns/etc... in later PRs.
>
> Thanks!
> Stephen
>
> [1] Testing IO transforms -
> https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing
>

-- 
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com