You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ismaël Mejía <ie...@gmail.com> on 2016/10/24 19:52:01 UTC
[DISCUSS] Current ongoing work on runners
Hello,
I am really happy to see new runners been contributed to our community
(e.g. GearPump and Apex recently). We have not discussed a lot about the
current capabilities of both runners.
Following the recent discussion about making ongoing work more explicit in
the mailing list, I would like to ask the people involved about the current
status of them, I think it is important to discuss this (apart of creating
the given JIRAs + updating the capability matrix docs) because more people
can eventually jump and give a hand on open issues.
I remember there was a google doc for the capabilities of each runner, is
this doc still available (sorry I lost the link). I suppose that once these
ongoing runners mature we can add this doc also to the website.
https://beam.apache.org/learn/runners/capability-matrix/
Regards,
Ismaël
ps. @Amit, given that the spark 2 (Dataset based) runner has also a feature
branch, if you consider it worth, can you please share a bit about that
work too.
ps2. Can anyone please share the link to the google doc I was talking
about, I can't find it after the recent changes to the website.
Re: [DISCUSS] Current ongoing work on runners
Posted by Frances Perry <fj...@google.com.INVALID>.
On Mon, Oct 24, 2016 at 12:52 PM, Ismaël Mejía <ie...@gmail.com> wrote:
> Hello,
>
> I am really happy to see new runners been contributed to our community
> (e.g. GearPump and Apex recently). We have not discussed a lot about the
> current capabilities of both runners.
>
> Following the recent discussion about making ongoing work more explicit in
> the mailing list, I would like to ask the people involved about the current
> status of them, I think it is important to discuss this (apart of creating
> the given JIRAs + updating the capability matrix docs) because more people
> can eventually jump and give a hand on open issues.
>
Note that we recently added a list of current features branches (and their
jira components) here:
http://beam.incubator.apache.org/contribute/work-in-progress/
I agree it would be great to hear how each of them are progressing!
>
> I remember there was a google doc for the capabilities of each runner, is
> this doc still available (sorry I lost the link). I suppose that once these
> ongoing runners mature we can add this doc also to the website.
> https://beam.apache.org/learn/runners/capability-matrix/
That doc for the capability matrix was only used while we figured out the
initial data -- it's been fully replaced by the link you have. (Generally
information is moved to the website once it stabilizes.)
Adding information to this page is currently part of the requirements for
moving to the master branch:
http://beam.incubator.apache.org/contribute/contribution-guide/#merging-into-master
It'd be great to have this information available somewhere for each runner
on a feature branch too -- both what's currently supported and what is
expected to be supported before merging.
>
> Regards,
> Ismaël
>
> ps. @Amit, given that the spark 2 (Dataset based) runner has also a feature
> branch, if you consider it worth, can you please share a bit about that
> work too.
>
> ps2. Can anyone please share the link to the google doc I was talking
> about, I can't find it after the recent changes to the website.
>
Re: [DISCUSS] Current ongoing work on runners
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Good idea.
Regards
JB
\u2063\u200b
On Oct 25, 2016, 17:57, at 17:57, Aljoscha Krettek <al...@apache.org> wrote:
>I think we might need to update the capability matrix with some of the
>new
>features that have popped up. Immediate things that come to mind are:
>* Timer/State API for user DoFns (coupled with new-style DoFn) (not yet
>completely in master)
> * SplittableDoFn
>
>This would allow tracking the process in each of these for each runner
>and
>would not require hunting for that information in email threads.
>
>On Tue, 25 Oct 2016 at 08:12 Jean-Baptiste Onofr� <jb...@nanthrax.net>
>wrote:
>
>> +1. For me it's one of the most important point for the new website.
>We
>> should give a clear and exhaustive list of what we have, both for
>runners
>> and IOs (with supported features).
>>
>> Regards
>> JB
>>
>> \u2063\u200b
>>
>> On Oct 24, 2016, 21:52, at 21:52, "Isma�l Mej�a" <ie...@gmail.com>
>> wrote:
>> >Hello,
>> >
>> >I am really happy to see new runners been contributed to our
>community
>> >(e.g. GearPump and Apex recently). We have not discussed a lot about
>> >the
>> >current capabilities of both runners.
>> >
>> >Following the recent discussion about making ongoing work more
>explicit
>> >in
>> >the mailing list, I would like to ask the people involved about the
>> >current
>> >status of them, I think it is important to discuss this (apart of
>> >creating
>> >the given JIRAs + updating the capability matrix docs) because more
>> >people
>> >can eventually jump and give a hand on open issues.
>> >
>> >I remember there was a google doc for the capabilities of each
>runner,
>> >is
>> >this doc still available (sorry I lost the link). I suppose that
>once
>> >these
>> >ongoing runners mature we can add this doc also to the website.
>> >https://beam.apache.org/learn/runners/capability-matrix/
>> >
>> >Regards,
>> >Isma�l
>> >
>> >ps. @Amit, given that the spark 2 (Dataset based) runner has also a
>> >feature
>> >branch, if you consider it worth, can you please share a bit about
>that
>> >work too.
>> >
>> >ps2. Can anyone please share the link to the google doc I was
>talking
>> >about, I can't find it after the recent changes to the website.
>> >\u200b
>>
Re: [DISCUSS] Current ongoing work on runners
Posted by Manu Zhang <ow...@gmail.com>.
We usually have docs sitting together with the source codes such that each
release has its own versioned docs. If the capability matrix is like other
codes, we can update it as we add new features. It also applies to other
docs like new IO. We could make it a requirement in the PR template.
Thanks,
Manu
On Wed, Oct 26, 2016 at 7:24 AM Thomas Weise <th...@apache.org> wrote:
> I'm planning to take up the discussion about Apex runner current state and
> proposed next steps in a separate thread.
>
> Thanks,
> Thomas
>
>
> On Tue, Oct 25, 2016 at 10:32 AM, Amit Sela <am...@gmail.com> wrote:
>
> > SparkRunner status:
> >
> > V1 (Spark 1.6.x - DStream/RDD API):
> > *Batch*: Full model support for batch, continuous ROS testing setup is in
> > process now so that CI will validate constantly.
> > *Streaming*: Supporting UnboundedSource is in review
> > <https://github.com/apache/incubator-beam/pull/1143>, starting to work
> on
> > triggers and accumulation modes now.
> >
> > V2 (Spark 2.x - Dataset API):
> > This is on hold for now as Spark 2.0 - Dataset AP for streaming (AKA
> > "Structured Streaming") is marked Alpha.
> > In addition, there are still some basic properties in the Dataset API
> that
> > are missing and will be required to properly support Beam:
> >
> > - Stateful operators.
> > - Encoders (Spark's new schema-based coders) optimization support for
> > classes that are a bit more sophisticated than POJO's (generics, inner
> > classes, etc.).
> >
> > Also waiting to see if Watermarks and purging late/stale data will be
> > introduced in 2.1 (currently the Dataset grows indefinitely which is not
> > something acceptable for production applications).
> > Once this becomes more clear (2.1 release ?) I will get back to working
> on
> > this because in general the Dataset API is preferred as it is actually a
> > real unified API for batch and streaming (and the schema-based
> > optimizations are also interesting).
> >
> > I hope this gives a clear view of the SparkRunner status, feel free to
> ping
> > me for more details on the user/dev list or Slack.
> >
> > Thanks,
> > Amit
> >
> > On Tue, Oct 25, 2016 at 6:57 PM Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> > > I think we might need to update the capability matrix with some of the
> > new
> > > features that have popped up. Immediate things that come to mind are:
> > > * Timer/State API for user DoFns (coupled with new-style DoFn) (not
> yet
> > > completely in master)
> > > * SplittableDoFn
> > >
> > > This would allow tracking the process in each of these for each runner
> > and
> > > would not require hunting for that information in email threads.
> > >
> > > On Tue, 25 Oct 2016 at 08:12 Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > >
> > > > +1. For me it's one of the most important point for the new website.
> We
> > > > should give a clear and exhaustive list of what we have, both for
> > runners
> > > > and IOs (with supported features).
> > > >
> > > > Regards
> > > > JB
> > > >
> > > >
> > > >
> > > > On Oct 24, 2016, 21:52, at 21:52, "Ismaël Mejía" <ie...@gmail.com>
> > > > wrote:
> > > > >Hello,
> > > > >
> > > > >I am really happy to see new runners been contributed to our
> community
> > > > >(e.g. GearPump and Apex recently). We have not discussed a lot about
> > > > >the
> > > > >current capabilities of both runners.
> > > > >
> > > > >Following the recent discussion about making ongoing work more
> > explicit
> > > > >in
> > > > >the mailing list, I would like to ask the people involved about the
> > > > >current
> > > > >status of them, I think it is important to discuss this (apart of
> > > > >creating
> > > > >the given JIRAs + updating the capability matrix docs) because more
> > > > >people
> > > > >can eventually jump and give a hand on open issues.
> > > > >
> > > > >I remember there was a google doc for the capabilities of each
> > runner,
> > > > >is
> > > > >this doc still available (sorry I lost the link). I suppose that
> once
> > > > >these
> > > > >ongoing runners mature we can add this doc also to the website.
> > > > >https://beam.apache.org/learn/runners/capability-matrix/
> > > > >
> > > > >Regards,
> > > > >Ismaël
> > > > >
> > > > >ps. @Amit, given that the spark 2 (Dataset based) runner has also a
> > > > >feature
> > > > >branch, if you consider it worth, can you please share a bit about
> > that
> > > > >work too.
> > > > >
> > > > >ps2. Can anyone please share the link to the google doc I was
> talking
> > > > >about, I can't find it after the recent changes to the website.
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Current ongoing work on runners
Posted by Thomas Weise <th...@apache.org>.
I'm planning to take up the discussion about Apex runner current state and
proposed next steps in a separate thread.
Thanks,
Thomas
On Tue, Oct 25, 2016 at 10:32 AM, Amit Sela <am...@gmail.com> wrote:
> SparkRunner status:
>
> V1 (Spark 1.6.x - DStream/RDD API):
> *Batch*: Full model support for batch, continuous ROS testing setup is in
> process now so that CI will validate constantly.
> *Streaming*: Supporting UnboundedSource is in review
> <https://github.com/apache/incubator-beam/pull/1143>, starting to work on
> triggers and accumulation modes now.
>
> V2 (Spark 2.x - Dataset API):
> This is on hold for now as Spark 2.0 - Dataset AP for streaming (AKA
> "Structured Streaming") is marked Alpha.
> In addition, there are still some basic properties in the Dataset API that
> are missing and will be required to properly support Beam:
>
> - Stateful operators.
> - Encoders (Spark's new schema-based coders) optimization support for
> classes that are a bit more sophisticated than POJO's (generics, inner
> classes, etc.).
>
> Also waiting to see if Watermarks and purging late/stale data will be
> introduced in 2.1 (currently the Dataset grows indefinitely which is not
> something acceptable for production applications).
> Once this becomes more clear (2.1 release ?) I will get back to working on
> this because in general the Dataset API is preferred as it is actually a
> real unified API for batch and streaming (and the schema-based
> optimizations are also interesting).
>
> I hope this gives a clear view of the SparkRunner status, feel free to ping
> me for more details on the user/dev list or Slack.
>
> Thanks,
> Amit
>
> On Tue, Oct 25, 2016 at 6:57 PM Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > I think we might need to update the capability matrix with some of the
> new
> > features that have popped up. Immediate things that come to mind are:
> > * Timer/State API for user DoFns (coupled with new-style DoFn) (not yet
> > completely in master)
> > * SplittableDoFn
> >
> > This would allow tracking the process in each of these for each runner
> and
> > would not require hunting for that information in email threads.
> >
> > On Tue, 25 Oct 2016 at 08:12 Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> > > +1. For me it's one of the most important point for the new website. We
> > > should give a clear and exhaustive list of what we have, both for
> runners
> > > and IOs (with supported features).
> > >
> > > Regards
> > > JB
> > >
> > >
> > >
> > > On Oct 24, 2016, 21:52, at 21:52, "Ismaël Mejía" <ie...@gmail.com>
> > > wrote:
> > > >Hello,
> > > >
> > > >I am really happy to see new runners been contributed to our community
> > > >(e.g. GearPump and Apex recently). We have not discussed a lot about
> > > >the
> > > >current capabilities of both runners.
> > > >
> > > >Following the recent discussion about making ongoing work more
> explicit
> > > >in
> > > >the mailing list, I would like to ask the people involved about the
> > > >current
> > > >status of them, I think it is important to discuss this (apart of
> > > >creating
> > > >the given JIRAs + updating the capability matrix docs) because more
> > > >people
> > > >can eventually jump and give a hand on open issues.
> > > >
> > > >I remember there was a google doc for the capabilities of each
> runner,
> > > >is
> > > >this doc still available (sorry I lost the link). I suppose that once
> > > >these
> > > >ongoing runners mature we can add this doc also to the website.
> > > >https://beam.apache.org/learn/runners/capability-matrix/
> > > >
> > > >Regards,
> > > >Ismaël
> > > >
> > > >ps. @Amit, given that the spark 2 (Dataset based) runner has also a
> > > >feature
> > > >branch, if you consider it worth, can you please share a bit about
> that
> > > >work too.
> > > >
> > > >ps2. Can anyone please share the link to the google doc I was talking
> > > >about, I can't find it after the recent changes to the website.
> > > >
> > >
> >
>
Re: [DISCUSS] Current ongoing work on runners
Posted by Amit Sela <am...@gmail.com>.
SparkRunner status:
V1 (Spark 1.6.x - DStream/RDD API):
*Batch*: Full model support for batch, continuous ROS testing setup is in
process now so that CI will validate constantly.
*Streaming*: Supporting UnboundedSource is in review
<https://github.com/apache/incubator-beam/pull/1143>, starting to work on
triggers and accumulation modes now.
V2 (Spark 2.x - Dataset API):
This is on hold for now as Spark 2.0 - Dataset AP for streaming (AKA
"Structured Streaming") is marked Alpha.
In addition, there are still some basic properties in the Dataset API that
are missing and will be required to properly support Beam:
- Stateful operators.
- Encoders (Spark's new schema-based coders) optimization support for
classes that are a bit more sophisticated than POJO's (generics, inner
classes, etc.).
Also waiting to see if Watermarks and purging late/stale data will be
introduced in 2.1 (currently the Dataset grows indefinitely which is not
something acceptable for production applications).
Once this becomes more clear (2.1 release ?) I will get back to working on
this because in general the Dataset API is preferred as it is actually a
real unified API for batch and streaming (and the schema-based
optimizations are also interesting).
I hope this gives a clear view of the SparkRunner status, feel free to ping
me for more details on the user/dev list or Slack.
Thanks,
Amit
On Tue, Oct 25, 2016 at 6:57 PM Aljoscha Krettek <al...@apache.org>
wrote:
> I think we might need to update the capability matrix with some of the new
> features that have popped up. Immediate things that come to mind are:
> * Timer/State API for user DoFns (coupled with new-style DoFn) (not yet
> completely in master)
> * SplittableDoFn
>
> This would allow tracking the process in each of these for each runner and
> would not require hunting for that information in email threads.
>
> On Tue, 25 Oct 2016 at 08:12 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> > +1. For me it's one of the most important point for the new website. We
> > should give a clear and exhaustive list of what we have, both for runners
> > and IOs (with supported features).
> >
> > Regards
> > JB
> >
> >
> >
> > On Oct 24, 2016, 21:52, at 21:52, "Ismaël Mejía" <ie...@gmail.com>
> > wrote:
> > >Hello,
> > >
> > >I am really happy to see new runners been contributed to our community
> > >(e.g. GearPump and Apex recently). We have not discussed a lot about
> > >the
> > >current capabilities of both runners.
> > >
> > >Following the recent discussion about making ongoing work more explicit
> > >in
> > >the mailing list, I would like to ask the people involved about the
> > >current
> > >status of them, I think it is important to discuss this (apart of
> > >creating
> > >the given JIRAs + updating the capability matrix docs) because more
> > >people
> > >can eventually jump and give a hand on open issues.
> > >
> > >I remember there was a google doc for the capabilities of each runner,
> > >is
> > >this doc still available (sorry I lost the link). I suppose that once
> > >these
> > >ongoing runners mature we can add this doc also to the website.
> > >https://beam.apache.org/learn/runners/capability-matrix/
> > >
> > >Regards,
> > >Ismaël
> > >
> > >ps. @Amit, given that the spark 2 (Dataset based) runner has also a
> > >feature
> > >branch, if you consider it worth, can you please share a bit about that
> > >work too.
> > >
> > >ps2. Can anyone please share the link to the google doc I was talking
> > >about, I can't find it after the recent changes to the website.
> > >
> >
>
Re: [DISCUSS] Current ongoing work on runners
Posted by Aljoscha Krettek <al...@apache.org>.
I think we might need to update the capability matrix with some of the new
features that have popped up. Immediate things that come to mind are:
* Timer/State API for user DoFns (coupled with new-style DoFn) (not yet
completely in master)
* SplittableDoFn
This would allow tracking the process in each of these for each runner and
would not require hunting for that information in email threads.
On Tue, 25 Oct 2016 at 08:12 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> +1. For me it's one of the most important point for the new website. We
> should give a clear and exhaustive list of what we have, both for runners
> and IOs (with supported features).
>
> Regards
> JB
>
>
>
> On Oct 24, 2016, 21:52, at 21:52, "Ismaël Mejía" <ie...@gmail.com>
> wrote:
> >Hello,
> >
> >I am really happy to see new runners been contributed to our community
> >(e.g. GearPump and Apex recently). We have not discussed a lot about
> >the
> >current capabilities of both runners.
> >
> >Following the recent discussion about making ongoing work more explicit
> >in
> >the mailing list, I would like to ask the people involved about the
> >current
> >status of them, I think it is important to discuss this (apart of
> >creating
> >the given JIRAs + updating the capability matrix docs) because more
> >people
> >can eventually jump and give a hand on open issues.
> >
> >I remember there was a google doc for the capabilities of each runner,
> >is
> >this doc still available (sorry I lost the link). I suppose that once
> >these
> >ongoing runners mature we can add this doc also to the website.
> >https://beam.apache.org/learn/runners/capability-matrix/
> >
> >Regards,
> >Ismaël
> >
> >ps. @Amit, given that the spark 2 (Dataset based) runner has also a
> >feature
> >branch, if you consider it worth, can you please share a bit about that
> >work too.
> >
> >ps2. Can anyone please share the link to the google doc I was talking
> >about, I can't find it after the recent changes to the website.
> >
>
Re: [DISCUSS] Current ongoing work on runners
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1. For me it's one of the most important point for the new website. We should give a clear and exhaustive list of what we have, both for runners and IOs (with supported features).
Regards
JB
\u2063\u200b
On Oct 24, 2016, 21:52, at 21:52, "Isma�l Mej�a" <ie...@gmail.com> wrote:
>Hello,
>
>I am really happy to see new runners been contributed to our community
>(e.g. GearPump and Apex recently). We have not discussed a lot about
>the
>current capabilities of both runners.
>
>Following the recent discussion about making ongoing work more explicit
>in
>the mailing list, I would like to ask the people involved about the
>current
>status of them, I think it is important to discuss this (apart of
>creating
>the given JIRAs + updating the capability matrix docs) because more
>people
>can eventually jump and give a hand on open issues.
>
>I remember there was a google doc for the capabilities of each runner,
>is
>this doc still available (sorry I lost the link). I suppose that once
>these
>ongoing runners mature we can add this doc also to the website.
>https://beam.apache.org/learn/runners/capability-matrix/
>
>Regards,
>Isma�l
>
>ps. @Amit, given that the spark 2 (Dataset based) runner has also a
>feature
>branch, if you consider it worth, can you please share a bit about that
>work too.
>
>ps2. Can anyone please share the link to the google doc I was talking
>about, I can't find it after the recent changes to the website.
>\u200b
Re: [DISCUSS] Current ongoing work on runners
Posted by Robert Bradshaw <ro...@google.com.INVALID>.
I think it would be worth publishing a compatibility matrix, if not on
the main site, as part of the branch itself.
Even better would be if the compatibility matrix was automatically
deduced based on a suite of tests that each runner could (attempt to)
pass.
On Mon, Oct 24, 2016 at 12:52 PM, Ismaël Mejía <ie...@gmail.com> wrote:
> Hello,
>
> I am really happy to see new runners been contributed to our community
> (e.g. GearPump and Apex recently). We have not discussed a lot about the
> current capabilities of both runners.
>
> Following the recent discussion about making ongoing work more explicit in
> the mailing list, I would like to ask the people involved about the current
> status of them, I think it is important to discuss this (apart of creating
> the given JIRAs + updating the capability matrix docs) because more people
> can eventually jump and give a hand on open issues.
>
> I remember there was a google doc for the capabilities of each runner, is
> this doc still available (sorry I lost the link). I suppose that once these
> ongoing runners mature we can add this doc also to the website.
> https://beam.apache.org/learn/runners/capability-matrix/
>
> Regards,
> Ismaël
>
> ps. @Amit, given that the spark 2 (Dataset based) runner has also a feature
> branch, if you consider it worth, can you please share a bit about that
> work too.
>
> ps2. Can anyone please share the link to the google doc I was talking
> about, I can't find it after the recent changes to the website.
>