You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Tao Meng <oa...@gmail.com> on 2017/01/09 15:34:34 UTC

Can beam create a iterative streaming job?

Hi all,
  I have a question about iterative streaming job. Can beam create a
iterative streaming job? For example,
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/iteration/IterateExample.java

Thanks

Re: Can beam create a iterative streaming job?

Posted by Tao Meng <oa...@gmail.com>.

Thanks

Lukasz Cwik <lc...@google.com>于2017年1月10日周二 上午1:54写道：

> There are no built in constructs which allow for a conditional iterative
> execution. This issue (https://issues.apache.org/jira/browse/BEAM-106)
> tracks the feature request.
> Today, if you know some upperbound on how many times your pipeline needs
> to loop then you can manually unroll the loop.
> Alternatively you can create a cycle by reading and writing to the same
> unbounded source/sink (e.g. read and write to the same Kafka topic).
>
> On Mon, Jan 9, 2017 at 7:34 AM, Tao Meng <oa...@gmail.com> wrote:
>
> Hi all,
>   I have a question about iterative streaming job. Can beam create a
> iterative streaming job? For example,
> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/iteration/IterateExample.java
>
> Thanks
>
>
>

Re: Can beam create a iterative streaming job?

Posted by Lukasz Cwik <lc...@google.com>.

There are no built in constructs which allow for a conditional iterative
execution. This issue (https://issues.apache.org/jira/browse/BEAM-106)
tracks the feature request.
Today, if you know some upperbound on how many times your pipeline needs to
loop then you can manually unroll the loop.
Alternatively you can create a cycle by reading and writing to the same
unbounded source/sink (e.g. read and write to the same Kafka topic).

On Mon, Jan 9, 2017 at 7:34 AM, Tao Meng <oa...@gmail.com> wrote:

> Hi all,
>   I have a question about iterative streaming job. Can beam create a
> iterative streaming job? For example, https://github.com/
> apache/flink/blob/master/flink-examples/flink-examples-
> streaming/src/main/java/org/apache/flink/streaming/examples/iteration/
> IterateExample.java
>
> Thanks
>