You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by 송원욱 <wo...@apache.org> on 2018/11/16 05:51:56 UTC
A new Beam Runner on Apache Nemo
Hello all!
I'm a member of the Apache Nemo community, another Apache project for
processing big data focusing on easy-to-use, flexible optimizations for
various deployment environments. More information can be seen on our website
<http://nemo.apache.org>. We've been building the system for quite a while
now, and we have been using Apache Beam as one of the programming layers
that we support for writing data processing applications. We have already
taken a look at the capability matrix
<https://beam.apache.org/documentation/runners/capability-matrix/> of Beam
runners, and the runner authoring guide
<https://beam.apache.org/contribute/runner-guide/>, and we have been
successful in implementing a large portion of the capability criteria.
With the progress, we wish to be able to list our runner as one of the Beam
runners, to be able to notify the users that our system supports Beam, and
that Beam users have another option to choose from for running their data
processing applications. It would be lovely to know the details of the
process required for it!
Thanks!
Wonook
Re: A new Beam Runner on Apache Nemo
Posted by 송원욱 <wo...@apache.org>.
It's been a while, but just to let you know that there's a PR up regarding
the issue! Anyone who's interested can take a look at
https://github.com/apache/beam/pull/7236.
Wonook
2018년 11월 19일 (월) 오후 2:12, 송원욱 <wo...@apache.org>님이 작성:
> Thanks for the reply and the help!
>
> At the moment, we are thinking about keeping the Runner outside Beam for
> the time being, as there are a number of extra ongoing developments going
> regarding a few features for stream processing. I'll submit a PR for the
> website in a short time with the details for the capability matrix, and on
> how to use our Runner, with external links and references, etc.
>
> Regarding the portability layer, at the moment we have been focusing on
> supporting the various features supported by the Java Beam SDK on our
> system and improving the performance of the system itself, but we will
> definitely work on the portability layer with the Nemo Runner, as it sounds
> more than exciting to be able to run Python programs on Apache Nemo. I'll
> definitely check up on the ValidatesRunner tests. We have been running our
> tests as well, so we are quite confident that it would run without much
> problems. Thanks for the tip for the Maven project!
>
> Thanks,
> Wonook
>
>
> 2018년 11월 17일 (토) 오전 1:48, Kenneth Knowles <ke...@apache.org>님이 작성:
>
>> Hi Wonook,
>>
>> Very cool! I see it here:
>> https://github.com/apache/incubator-nemo/tree/master/compiler/frontend/beam/src/main/java/org/apache/nemo/compiler/frontend/beam
>>
>> Some more details on what Max said about running the ValidatesRunner
>> tests:
>>
>> - if you are planning to contribute the runner to Beam, you can use the
>> other runners as examples and generally the whole community is likely to
>> keep your config up to date
>>
>> - if you are planning to keep the runner as part of Apache Nemo, then I
>> see you are using Maven to build so you can use an old snapshot as an
>> example, like this:
>> https://github.com/apache/beam/blob/v2.4.0/runners/gearpump/pom.xml#L55
>>
>> Kenn
>>
>> On Fri, Nov 16, 2018 at 3:16 AM Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>>> Hi Wonook,
>>>
>>> First of all, welcome to the Beam community! It is great to see another
>>> Runner emerging.
>>>
>>> If you're planning to contribute your Runner to Beam, you should verify
>>> the compatibility with the ValidatesRunner integration tests. Then open
>>> a PR with documentation, a Runner page, and updates to the matrix.
>>>
>>> If you're planning to leave the Runner outside Beam for the time being,
>>> please submit a Runner page for the Beam website. The page should
>>> contain information on how to use the Runner and a link to the external
>>> web site with up-to-date information.
>>>
>>> Feel free to ask here or in our Slack channel if you have more questions.
>>>
>>> I'm also curious, have you looked into integrating portability with the
>>> Nemo Runner?
>>>
>>> Thanks,
>>> Max
>>>
>>> On 16.11.18 06:51, 송원욱 wrote:
>>> > Hello all!
>>> >
>>> > I'm a member of the Apache Nemo community, another Apache project for
>>> > processing big data focusing on easy-to-use, flexible optimizations
>>> for
>>> > various deployment environments. More information can be seen on our
>>> > website <http://nemo.apache.org>. We've been building the system for
>>> > quite a while now, and we have been using Apache Beam as one of the
>>> > programming layers that we support for writing data processing
>>> > applications. We have already taken a look at the capability matrix
>>> > <https://beam.apache.org/documentation/runners/capability-matrix/> of
>>> > Beam runners, and the runner authoring guide
>>> > <https://beam.apache.org/contribute/runner-guide/>, and we have been
>>> > successful in implementing a large portion of the capability criteria.
>>> >
>>> > With the progress, we wish to be able to list our runner as one of the
>>> > Beam runners, to be able to notify the users that our system supports
>>> > Beam, and that Beam users have another option to choose from for
>>> running
>>> > their data processing applications. It would be lovely to know the
>>> > details of the process required for it!
>>> >
>>> > Thanks!
>>> >
>>> >
>>> > Wonook
>>> >
>>>
>>
Re: A new Beam Runner on Apache Nemo
Posted by 송원욱 <wo...@apache.org>.
Thanks for the reply and the help!
At the moment, we are thinking about keeping the Runner outside Beam for
the time being, as there are a number of extra ongoing developments going
regarding a few features for stream processing. I'll submit a PR for the
website in a short time with the details for the capability matrix, and on
how to use our Runner, with external links and references, etc.
Regarding the portability layer, at the moment we have been focusing on
supporting the various features supported by the Java Beam SDK on our
system and improving the performance of the system itself, but we will
definitely work on the portability layer with the Nemo Runner, as it sounds
more than exciting to be able to run Python programs on Apache Nemo. I'll
definitely check up on the ValidatesRunner tests. We have been running our
tests as well, so we are quite confident that it would run without much
problems. Thanks for the tip for the Maven project!
Thanks,
Wonook
2018년 11월 17일 (토) 오전 1:48, Kenneth Knowles <ke...@apache.org>님이 작성:
> Hi Wonook,
>
> Very cool! I see it here:
> https://github.com/apache/incubator-nemo/tree/master/compiler/frontend/beam/src/main/java/org/apache/nemo/compiler/frontend/beam
>
> Some more details on what Max said about running the ValidatesRunner tests:
>
> - if you are planning to contribute the runner to Beam, you can use the
> other runners as examples and generally the whole community is likely to
> keep your config up to date
>
> - if you are planning to keep the runner as part of Apache Nemo, then I
> see you are using Maven to build so you can use an old snapshot as an
> example, like this:
> https://github.com/apache/beam/blob/v2.4.0/runners/gearpump/pom.xml#L55
>
> Kenn
>
> On Fri, Nov 16, 2018 at 3:16 AM Maximilian Michels <mx...@apache.org> wrote:
>
>> Hi Wonook,
>>
>> First of all, welcome to the Beam community! It is great to see another
>> Runner emerging.
>>
>> If you're planning to contribute your Runner to Beam, you should verify
>> the compatibility with the ValidatesRunner integration tests. Then open
>> a PR with documentation, a Runner page, and updates to the matrix.
>>
>> If you're planning to leave the Runner outside Beam for the time being,
>> please submit a Runner page for the Beam website. The page should
>> contain information on how to use the Runner and a link to the external
>> web site with up-to-date information.
>>
>> Feel free to ask here or in our Slack channel if you have more questions.
>>
>> I'm also curious, have you looked into integrating portability with the
>> Nemo Runner?
>>
>> Thanks,
>> Max
>>
>> On 16.11.18 06:51, 송원욱 wrote:
>> > Hello all!
>> >
>> > I'm a member of the Apache Nemo community, another Apache project for
>> > processing big data focusing on easy-to-use, flexible optimizations for
>> > various deployment environments. More information can be seen on our
>> > website <http://nemo.apache.org>. We've been building the system for
>> > quite a while now, and we have been using Apache Beam as one of the
>> > programming layers that we support for writing data processing
>> > applications. We have already taken a look at the capability matrix
>> > <https://beam.apache.org/documentation/runners/capability-matrix/> of
>> > Beam runners, and the runner authoring guide
>> > <https://beam.apache.org/contribute/runner-guide/>, and we have been
>> > successful in implementing a large portion of the capability criteria.
>> >
>> > With the progress, we wish to be able to list our runner as one of the
>> > Beam runners, to be able to notify the users that our system supports
>> > Beam, and that Beam users have another option to choose from for
>> running
>> > their data processing applications. It would be lovely to know the
>> > details of the process required for it!
>> >
>> > Thanks!
>> >
>> >
>> > Wonook
>> >
>>
>
Re: A new Beam Runner on Apache Nemo
Posted by Kenneth Knowles <ke...@apache.org>.
Hi Wonook,
Very cool! I see it here:
https://github.com/apache/incubator-nemo/tree/master/compiler/frontend/beam/src/main/java/org/apache/nemo/compiler/frontend/beam
Some more details on what Max said about running the ValidatesRunner tests:
- if you are planning to contribute the runner to Beam, you can use the
other runners as examples and generally the whole community is likely to
keep your config up to date
- if you are planning to keep the runner as part of Apache Nemo, then I
see you are using Maven to build so you can use an old snapshot as an
example, like this:
https://github.com/apache/beam/blob/v2.4.0/runners/gearpump/pom.xml#L55
Kenn
On Fri, Nov 16, 2018 at 3:16 AM Maximilian Michels <mx...@apache.org> wrote:
> Hi Wonook,
>
> First of all, welcome to the Beam community! It is great to see another
> Runner emerging.
>
> If you're planning to contribute your Runner to Beam, you should verify
> the compatibility with the ValidatesRunner integration tests. Then open
> a PR with documentation, a Runner page, and updates to the matrix.
>
> If you're planning to leave the Runner outside Beam for the time being,
> please submit a Runner page for the Beam website. The page should
> contain information on how to use the Runner and a link to the external
> web site with up-to-date information.
>
> Feel free to ask here or in our Slack channel if you have more questions.
>
> I'm also curious, have you looked into integrating portability with the
> Nemo Runner?
>
> Thanks,
> Max
>
> On 16.11.18 06:51, 송원욱 wrote:
> > Hello all!
> >
> > I'm a member of the Apache Nemo community, another Apache project for
> > processing big data focusing on easy-to-use, flexible optimizations for
> > various deployment environments. More information can be seen on our
> > website <http://nemo.apache.org>. We've been building the system for
> > quite a while now, and we have been using Apache Beam as one of the
> > programming layers that we support for writing data processing
> > applications. We have already taken a look at the capability matrix
> > <https://beam.apache.org/documentation/runners/capability-matrix/> of
> > Beam runners, and the runner authoring guide
> > <https://beam.apache.org/contribute/runner-guide/>, and we have been
> > successful in implementing a large portion of the capability criteria.
> >
> > With the progress, we wish to be able to list our runner as one of the
> > Beam runners, to be able to notify the users that our system supports
> > Beam, and that Beam users have another option to choose from for running
> > their data processing applications. It would be lovely to know the
> > details of the process required for it!
> >
> > Thanks!
> >
> >
> > Wonook
> >
>
Re: A new Beam Runner on Apache Nemo
Posted by Maximilian Michels <mx...@apache.org>.
Hi Wonook,
First of all, welcome to the Beam community! It is great to see another
Runner emerging.
If you're planning to contribute your Runner to Beam, you should verify
the compatibility with the ValidatesRunner integration tests. Then open
a PR with documentation, a Runner page, and updates to the matrix.
If you're planning to leave the Runner outside Beam for the time being,
please submit a Runner page for the Beam website. The page should
contain information on how to use the Runner and a link to the external
web site with up-to-date information.
Feel free to ask here or in our Slack channel if you have more questions.
I'm also curious, have you looked into integrating portability with the
Nemo Runner?
Thanks,
Max
On 16.11.18 06:51, 송원욱 wrote:
> Hello all!
>
> I'm a member of the Apache Nemo community, another Apache project for
> processing big data focusing on easy-to-use, flexible optimizations for
> various deployment environments. More information can be seen on our
> website <http://nemo.apache.org>. We've been building the system for
> quite a while now, and we have been using Apache Beam as one of the
> programming layers that we support for writing data processing
> applications. We have already taken a look at the capability matrix
> <https://beam.apache.org/documentation/runners/capability-matrix/> of
> Beam runners, and the runner authoring guide
> <https://beam.apache.org/contribute/runner-guide/>, and we have been
> successful in implementing a large portion of the capability criteria.
>
> With the progress, we wish to be able to list our runner as one of the
> Beam runners, to be able to notify the users that our system supports
> Beam, and that Beam users have another option to choose from for running
> their data processing applications. It would be lovely to know the
> details of the process required for it!
>
> Thanks!
>
>
> Wonook
>