You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by 송원욱 <ws...@gmail.com> on 2017/04/13 06:53:10 UTC

Question regarding loops in BEAM programs

Hi,

I have been using BEAM to express different workloads to run them using
different runners. While doing this, I have encountered a question
regarding expression of workloads with multiple iterations of operators (ML
applications like ALS, MLR). As far as I know, it is currently not
supported to express loops with BEAM programs, so they are simply
'unrolled' to be expressed as an acyclic graph. However, would it be
possible to recognize operators that are inside loops from other operators
that are outside of those loops? If not, would it be possible to tag them
somehow to express them from others? Any other suggestions or plans for
carrying out the approach are welcome.

Thanks,
Wonook

Re: Question regarding loops in BEAM programs

Posted by 송원욱 <ws...@gmail.com>.

Thanks for the insight! It seems very similar to what I have been trying to
do :)
Wonook


2017-04-13 16:20 GMT+09:00 Robert Bradshaw <ro...@google.com.invalid>:

> There is no way (short of inspecting stack traces and bytecodes) for
> Beam to distinguish between
>
>     for (int i=0; i<3; i++) {
>         pc = pc.apply(MyTransform());
>     }
>
> from
>
>     pc.apply(MyTransform()).apply(MyTransform()).apply(MyTransform());
>
> However, PTransforms are hierarchal, so you could put the entire loop
> inside an enclosing PTransform which would be good practice.
> https://cloud.google.com/dataflow/model/composite-transforms (in Beam
> one overrides the expand method; but
> https://issues.apache.org/jira/browse/BEAM-1452 ). The "tagging" that
> is done is that the full name of the inner transforms are all prefixed
> with the name of the outer transform.
>
>
> On Wed, Apr 12, 2017 at 11:53 PM, 송원욱 <ws...@gmail.com> wrote:
> > Hi,
> >
> > I have been using BEAM to express different workloads to run them using
> > different runners. While doing this, I have encountered a question
> > regarding expression of workloads with multiple iterations of operators
> (ML
> > applications like ALS, MLR). As far as I know, it is currently not
> > supported to express loops with BEAM programs, so they are simply
> > 'unrolled' to be expressed as an acyclic graph. However, would it be
> > possible to recognize operators that are inside loops from other
> operators
> > that are outside of those loops? If not, would it be possible to tag them
> > somehow to express them from others? Any other suggestions or plans for
> > carrying out the approach are welcome.
> >
> > Thanks,
> > Wonook
>

Re: Question regarding loops in BEAM programs

Posted by Robert Bradshaw <ro...@google.com.INVALID>.

There is no way (short of inspecting stack traces and bytecodes) for
Beam to distinguish between

    for (int i=0; i<3; i++) {
        pc = pc.apply(MyTransform());
    }

from

    pc.apply(MyTransform()).apply(MyTransform()).apply(MyTransform());

However, PTransforms are hierarchal, so you could put the entire loop
inside an enclosing PTransform which would be good practice.
https://cloud.google.com/dataflow/model/composite-transforms (in Beam
one overrides the expand method; but
https://issues.apache.org/jira/browse/BEAM-1452 ). The "tagging" that
is done is that the full name of the inner transforms are all prefixed
with the name of the outer transform.


On Wed, Apr 12, 2017 at 11:53 PM, 송원욱 <ws...@gmail.com> wrote:
> Hi,
>
> I have been using BEAM to express different workloads to run them using
> different runners. While doing this, I have encountered a question
> regarding expression of workloads with multiple iterations of operators (ML
> applications like ALS, MLR). As far as I know, it is currently not
> supported to express loops with BEAM programs, so they are simply
> 'unrolled' to be expressed as an acyclic graph. However, would it be
> possible to recognize operators that are inside loops from other operators
> that are outside of those loops? If not, would it be possible to tag them
> somehow to express them from others? Any other suggestions or plans for
> carrying out the approach are welcome.
>
> Thanks,
> Wonook