Re: [PROPOSAL] Pipeline Runner API design doc

[Aljoscha Krettek <al...@apache.org>]

Hi,
is there already a design doc on moving the triggers to be cross-language/a
syntax tree. It is mentioned in one line in this doc and in passing in some
other Jira issues and pull requests/commits. I'm quite interested in how
this would go. It would also not allow user-written triggers anymore,
correct?

Cheers,
Aljoscha

On Thu, 24 Mar 2016 at 07:41 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:

> Hi Kenn,
>
> thanks for the update. I'm reading it now.
>
> Regards
> JB
>
> On 03/23/2016 10:17 PM, Kenneth Knowles wrote:
> > Hi everyone,
> >
> > Incorporating the feedback from the 1-pager I circulated a week ago, I
> have
> > put together a concrete design document for the new API(s).
> >
> >
> https://docs.google.com/document/d/1bao-5B6uBuf-kwH1meenAuXXS0c9cBQ1B2J59I3FiyI/edit?usp=sharing
> >
> > I appreciate any and all feedback on the design.
> >
> > Kenn
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [PROPOSAL] Pipeline Runner API design doc

[Kenneth Knowles <kl...@google.com.INVALID>]

Hi Aljoscha,

Thanks for reading it over!

 - There is no trigger design doc circulating yet. You are right that it
would disable user-defined triggers against the current API. IMO the API is
too general/unstructured and tends to magnify errors. I do actually have a
doc on triggers that is the basis for making them language independent, but
I am still improving it to be useful to a broader audience.

 - We definitely want to keep ReduceFnRunner around for everyone to be able
to use, just more "under the hood". Hopefully as we refine things, it will
become less of a nexus of complexity.

 - Yes, I think we should definitely keep PIpelineVisitor, but add more
capabilities. This API can continue to be enhanced, I just tried to outline
in the doc the minimum for getting rid of the "apply" interception.

Kenn

On Mon, Apr 18, 2016 at 2:38 AM, Aljoscha Krettek <al...@apache.org>
wrote:

> Hi,
> also, I think the proposed changes are very good. I really like the clear
> separation of concerns imposed by the updated design.
>
> Speaking for the Flink runner, I think it should be very easy to migrate to
> the new scheme as long as something like the ReduceFnRunner is maintained
> for dealing with the low-level details of windows/triggers. Also, it's not
> mentioned in the doc, but transformation to a runner-specific job will
> still be handled by Pipeline.PipelineVisitor, right? Just wanted to make
> sure there.
>
> Cheers,
> Aljoscha
>
> On Sat, 16 Apr 2016 at 07:43 Aljoscha Krettek <al...@apache.org> wrote:
>
> > Hi,
> > is there already a design doc on moving the triggers to be
> > cross-language/a syntax tree. It is mentioned in one line in this doc and
> > in passing in some other Jira issues and pull requests/commits. I'm quite
> > interested in how this would go. It would also not allow user-written
> > triggers anymore, correct?
> >
> > Cheers,
> > Aljoscha
> >
> > On Thu, 24 Mar 2016 at 07:41 Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> >> Hi Kenn,
> >>
> >> thanks for the update. I'm reading it now.
> >>
> >> Regards
> >> JB
> >>
> >> On 03/23/2016 10:17 PM, Kenneth Knowles wrote:
> >> > Hi everyone,
> >> >
> >> > Incorporating the feedback from the 1-pager I circulated a week ago, I
> >> have
> >> > put together a concrete design document for the new API(s).
> >> >
> >> >
> >>
> https://docs.google.com/document/d/1bao-5B6uBuf-kwH1meenAuXXS0c9cBQ1B2J59I3FiyI/edit?usp=sharing
> >> >
> >> > I appreciate any and all feedback on the design.
> >> >
> >> > Kenn
> >> >
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
> >
>

Re: [PROPOSAL] Pipeline Runner API design doc

[Aljoscha Krettek <al...@apache.org>]

Hi,
also, I think the proposed changes are very good. I really like the clear
separation of concerns imposed by the updated design.

Speaking for the Flink runner, I think it should be very easy to migrate to
the new scheme as long as something like the ReduceFnRunner is maintained
for dealing with the low-level details of windows/triggers. Also, it's not
mentioned in the doc, but transformation to a runner-specific job will
still be handled by Pipeline.PipelineVisitor, right? Just wanted to make
sure there.

Cheers,
Aljoscha

On Sat, 16 Apr 2016 at 07:43 Aljoscha Krettek <al...@apache.org> wrote:

> Hi,
> is there already a design doc on moving the triggers to be
> cross-language/a syntax tree. It is mentioned in one line in this doc and
> in passing in some other Jira issues and pull requests/commits. I'm quite
> interested in how this would go. It would also not allow user-written
> triggers anymore, correct?
>
> Cheers,
> Aljoscha
>
> On Thu, 24 Mar 2016 at 07:41 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
>> Hi Kenn,
>>
>> thanks for the update. I'm reading it now.
>>
>> Regards
>> JB
>>
>> On 03/23/2016 10:17 PM, Kenneth Knowles wrote:
>> > Hi everyone,
>> >
>> > Incorporating the feedback from the 1-pager I circulated a week ago, I
>> have
>> > put together a concrete design document for the new API(s).
>> >
>> >
>> https://docs.google.com/document/d/1bao-5B6uBuf-kwH1meenAuXXS0c9cBQ1B2J59I3FiyI/edit?usp=sharing
>> >
>> > I appreciate any and all feedback on the design.
>> >
>> > Kenn
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>