You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Shen Li <cs...@gmail.com> on 2017/05/11 18:24:44 UTC
What is the use case and the expected behavior for a Flatten
transform with no input?
Hi,
The FlattenTest enforces that Flatten transform can handle empty input. Is
there any use case for this feature?
https://github.com/apache/beam/blob/release-2.0.0/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java#L128
How should a runner translate an empty Flatten? Should it be a
BoundedSource that emits a single watermark of value
BoundedWindow.TIMESTAMP_MAX_VALUE?
Thanks,
Shen
Re: What is the use case and the expected behavior for a Flatten
transform with no input?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
Hi Shen,
The most obvious use case to me would be a pipeline that is generated by a
program that loops out the input PCollections. Of course, it is quite easy
to work around this by adding an empty PCollection to the flatten, so I
don't think it is important.
Kenn
On Thu, May 11, 2017 at 11:24 AM, Shen Li <cs...@gmail.com> wrote:
> Hi,
>
> The FlattenTest enforces that Flatten transform can handle empty input. Is
> there any use case for this feature?
>
> https://github.com/apache/beam/blob/release-2.0.0/sdks/
> java/core/src/test/java/org/apache/beam/sdk/transforms/
> FlattenTest.java#L128
>
> How should a runner translate an empty Flatten? Should it be a
> BoundedSource that emits a single watermark of value
> BoundedWindow.TIMESTAMP_MAX_VALUE?
>
> Thanks,
>
> Shen
>
Re: What is the use case and the expected behavior for a Flatten
transform with no input?
Posted by Shen Li <cs...@gmail.com>.
Thanks!
Shen
On Thu, May 11, 2017 at 2:29 PM, Reuven Lax <re...@google.com.invalid>
wrote:
> We sometimes see this when users generate flattens with a for loop (e.g.
> over input sources for a side input). Sometimes there is no input, and the
> flatten is empty.
>
>
> On Thu, May 11, 2017 at 11:24 AM, Shen Li <cs...@gmail.com> wrote:
>
> > Hi,
> >
> > The FlattenTest enforces that Flatten transform can handle empty input.
> Is
> > there any use case for this feature?
> >
> > https://github.com/apache/beam/blob/release-2.0.0/sdks/
> > java/core/src/test/java/org/apache/beam/sdk/transforms/
> > FlattenTest.java#L128
> >
> > How should a runner translate an empty Flatten? Should it be a
> > BoundedSource that emits a single watermark of value
> > BoundedWindow.TIMESTAMP_MAX_VALUE?
> >
> > Thanks,
> >
> > Shen
> >
>
Re: What is the use case and the expected behavior for a Flatten
transform with no input?
Posted by Reuven Lax <re...@google.com.INVALID>.
We sometimes see this when users generate flattens with a for loop (e.g.
over input sources for a side input). Sometimes there is no input, and the
flatten is empty.
On Thu, May 11, 2017 at 11:24 AM, Shen Li <cs...@gmail.com> wrote:
> Hi,
>
> The FlattenTest enforces that Flatten transform can handle empty input. Is
> there any use case for this feature?
>
> https://github.com/apache/beam/blob/release-2.0.0/sdks/
> java/core/src/test/java/org/apache/beam/sdk/transforms/
> FlattenTest.java#L128
>
> How should a runner translate an empty Flatten? Should it be a
> BoundedSource that emits a single watermark of value
> BoundedWindow.TIMESTAMP_MAX_VALUE?
>
> Thanks,
>
> Shen
>