You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Kenneth Knowles <kl...@google.com.INVALID> on 2017/05/24 21:54:28 UTC
Re: Make runner implementation smoother?
Hi Romain,
These are great topics for discussion, and very good timing.
I strongly recommend joining dev@beam, as that is where most discussion
about implementing Beam runners takes place. On that list, I recently
shared a guide for implementing a Beam runner at
https://s.apache.org/beam-runner-guide. It will be improved and appear on
the Beam site fairly soon.
I have also taken the liberty of moving this discussion to dev@beam and
moved user@beam to BCC.
Kenn
On Wed, May 24, 2017 at 1:54 PM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Hi guys,
>
> congrats for the 2.0!
>
> I have a few question regarding a custom runner implementation, there is
> no particular order but adding numbers for later references:
>
> 1. why beam doesn't have (yet?) a RunnerAdapter with all primivite listed
> and enforced by API, if I got right the code (shout if not) each runner is
> creating its own processing context and convertion at the moment which kind
> of means beam doesn't abstract the runtime which makes it harder to enter
> into beam model IMHO (vs defining by contract all operations - potentially
> with defaults when compositions are possible)
> 2. close to previous point (still around runner): i find quite hard to
> browse the DAG of beam compared to a plain DAG, is it intended to be so low
> level, can't we get a simple graph model?
>
> Maybe I'm just hitting a not yet extended/defined land and therefore an
> user friendly API is missing or I missed a central concept - in this case
> shout :p.
>
> Any pointers would be very welcomed on how to implement a runner without
> redefining a full transpiler/converter - or is it outside beam scope?
>
>
> FYI here what I have https://github.com/rmannibucau/beam-hazelcast-runner
> and here where I'm stucked (hesitating to redefine a full transpiler since
> I expected beam to help): https://github.com/rmannibucau/beam-hazelcast-
> runner/blob/master/src/main/java/com/github/rmannibucau/
> beam/runner/hazelcast/HazelcastPipelineVisitor.java
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://blog-rmannibucau.rhcloud.com> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
> <https://javaeefactory-rmannibucau.rhcloud.com>
>
Re: Make runner implementation smoother?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
Yes, we have a Slack channel. I've sent you an invite.
Kenn
On Wed, May 24, 2017 at 3:33 PM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Hi Kenneth,
>
> thanks a lot, this doc is really helpful. Will try to move a bit forward
> in my tests when I'll get time (hopefully end of the week or next one).
>
> PS: saw some history on google but not sure what was the complete outcome:
> if there an IRC/slack/git-based chat for beam? Know ASF (infra) can provide
> #apache-beam on freenode
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://blog-rmannibucau.rhcloud.com> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
> <https://javaeefactory-rmannibucau.rhcloud.com>
>
> 2017-05-24 23:54 GMT+02:00 Kenneth Knowles <kl...@google.com>:
>
>> Hi Romain,
>>
>> These are great topics for discussion, and very good timing.
>>
>> I strongly recommend joining dev@beam, as that is where most discussion
>> about implementing Beam runners takes place. On that list, I recently
>> shared a guide for implementing a Beam runner at
>> https://s.apache.org/beam-runner-guide. It will be improved and appear
>> on the Beam site fairly soon.
>>
>> I have also taken the liberty of moving this discussion to dev@beam and
>> moved user@beam to BCC.
>>
>> Kenn
>>
>> On Wed, May 24, 2017 at 1:54 PM, Romain Manni-Bucau <
>> rmannibucau@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> congrats for the 2.0!
>>>
>>> I have a few question regarding a custom runner implementation, there is
>>> no particular order but adding numbers for later references:
>>>
>>> 1. why beam doesn't have (yet?) a RunnerAdapter with all primivite
>>> listed and enforced by API, if I got right the code (shout if not) each
>>> runner is creating its own processing context and convertion at the moment
>>> which kind of means beam doesn't abstract the runtime which makes it harder
>>> to enter into beam model IMHO (vs defining by contract all operations -
>>> potentially with defaults when compositions are possible)
>>> 2. close to previous point (still around runner): i find quite hard to
>>> browse the DAG of beam compared to a plain DAG, is it intended to be so low
>>> level, can't we get a simple graph model?
>>>
>>> Maybe I'm just hitting a not yet extended/defined land and therefore an
>>> user friendly API is missing or I missed a central concept - in this case
>>> shout :p.
>>>
>>> Any pointers would be very welcomed on how to implement a runner without
>>> redefining a full transpiler/converter - or is it outside beam scope?
>>>
>>>
>>> FYI here what I have https://github.com/rmanni
>>> bucau/beam-hazelcast-runner and here where I'm stucked (hesitating to
>>> redefine a full transpiler since I expected beam to help):
>>> https://github.com/rmannibucau/beam-hazelcast-runner/
>>> blob/master/src/main/java/com/github/rmannibucau/beam/runner
>>> /hazelcast/HazelcastPipelineVisitor.java
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://blog-rmannibucau.rhcloud.com> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
>>> <https://javaeefactory-rmannibucau.rhcloud.com>
>>>
>>
>>
>
Re: Make runner implementation smoother?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi Kenneth,
thanks a lot, this doc is really helpful. Will try to move a bit forward in
my tests when I'll get time (hopefully end of the week or next one).
PS: saw some history on google but not sure what was the complete outcome:
if there an IRC/slack/git-based chat for beam? Know ASF (infra) can provide
#apache-beam on freenode
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://blog-rmannibucau.rhcloud.com> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
<https://javaeefactory-rmannibucau.rhcloud.com>
2017-05-24 23:54 GMT+02:00 Kenneth Knowles <kl...@google.com>:
> Hi Romain,
>
> These are great topics for discussion, and very good timing.
>
> I strongly recommend joining dev@beam, as that is where most discussion
> about implementing Beam runners takes place. On that list, I recently
> shared a guide for implementing a Beam runner at
> https://s.apache.org/beam-runner-guide. It will be improved and appear on
> the Beam site fairly soon.
>
> I have also taken the liberty of moving this discussion to dev@beam and
> moved user@beam to BCC.
>
> Kenn
>
> On Wed, May 24, 2017 at 1:54 PM, Romain Manni-Bucau <rmannibucau@gmail.com
> > wrote:
>
>> Hi guys,
>>
>> congrats for the 2.0!
>>
>> I have a few question regarding a custom runner implementation, there is
>> no particular order but adding numbers for later references:
>>
>> 1. why beam doesn't have (yet?) a RunnerAdapter with all primivite listed
>> and enforced by API, if I got right the code (shout if not) each runner is
>> creating its own processing context and convertion at the moment which kind
>> of means beam doesn't abstract the runtime which makes it harder to enter
>> into beam model IMHO (vs defining by contract all operations - potentially
>> with defaults when compositions are possible)
>> 2. close to previous point (still around runner): i find quite hard to
>> browse the DAG of beam compared to a plain DAG, is it intended to be so low
>> level, can't we get a simple graph model?
>>
>> Maybe I'm just hitting a not yet extended/defined land and therefore an
>> user friendly API is missing or I missed a central concept - in this case
>> shout :p.
>>
>> Any pointers would be very welcomed on how to implement a runner without
>> redefining a full transpiler/converter - or is it outside beam scope?
>>
>>
>> FYI here what I have https://github.com/rmannibucau/beam-hazelcast-runner
>> and here where I'm stucked (hesitating to redefine a full transpiler since
>> I expected beam to help): https://github.com/rman
>> nibucau/beam-hazelcast-runner/blob/master/src/main/java/com/
>> github/rmannibucau/beam/runner/hazelcast/HazelcastPipelineVisitor.java
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://blog-rmannibucau.rhcloud.com> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
>> <https://javaeefactory-rmannibucau.rhcloud.com>
>>
>
>