You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Amit Sela <am...@gmail.com> on 2016/12/14 19:03:35 UTC
New testSideInputsWithMultipleWindows and should DoFnRunner explode
if DoFn contains a side input ?
Hi all,
Yesterday a new test was added to ParDoTest suite:
"testSideInputsWithMultipleWindows".
To the best of my understanding, it's meant to test sideInputs for elements
in multiple windows (unexploded).
The Spark runner uses the DoFnRunner (Simple) to process DoFns, and it will
explode compressed elements only if it's "tagged" as "Observes Window".
Should it also explode if it has sideInputs ?
Thanks,
Amit
Re: New testSideInputsWithMultipleWindows and should DoFnRunner
explode if DoFn contains a side input ?
Posted by Lukasz Cwik <lc...@google.com.INVALID>.
Yes, it should explode if it has side inputs as well.
On Wed, Dec 14, 2016 at 11:03 AM, Amit Sela <am...@gmail.com> wrote:
> Hi all,
>
> Yesterday a new test was added to ParDoTest suite:
> "testSideInputsWithMultipleWindows".
> To the best of my understanding, it's meant to test sideInputs for elements
> in multiple windows (unexploded).
>
> The Spark runner uses the DoFnRunner (Simple) to process DoFns, and it will
> explode compressed elements only if it's "tagged" as "Observes Window".
>
> Should it also explode if it has sideInputs ?
>
> Thanks,
> Amit
>
Re: New testSideInputsWithMultipleWindows and should DoFnRunner
explode if DoFn contains a side input ?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
Yes, this is a bug in SimplerDoFnRunner (or maybe some clarity on whether
or not it owns this) not the Spark runner. FWIW the test is definitely
correct, and runners-core has had this bug for a while. It is
https://issues.apache.org/jira/browse/BEAM-1149 and I'm on it.
On Wed, Dec 14, 2016 at 11:03 AM, Amit Sela <am...@gmail.com> wrote:
> Hi all,
>
> Yesterday a new test was added to ParDoTest suite:
> "testSideInputsWithMultipleWindows".
> To the best of my understanding, it's meant to test sideInputs for elements
> in multiple windows (unexploded).
>
> The Spark runner uses the DoFnRunner (Simple) to process DoFns, and it will
> explode compressed elements only if it's "tagged" as "Observes Window".
>
> Should it also explode if it has sideInputs ?
>
> Thanks,
> Amit
>