Hazelcast Jet Runner - validation tests

[Jozsef Bartok <jo...@hazelcast.com>]

Hi.

My name is Jozsef, I've been working on Runners based on Hazelcast Jet.
Plural because we have both an "old-style" and a "portable" Runner in
development (https://github.com/hazelcast/hazelcast-jet-beam-runner).

While our portable one isn't even functional yet, the "old-style" type of
Runner is a bit more mature. It handles only bounded data, but for that
case it passes all Beam tests of ValidatesRunner category and runs the
Nexmark suite successfully too (I'm refering only to correctness, because
performance is not yet where it can be, we aren't doing any Pipeline
surgery yet and no other optimizations either).

Since a few days we have started extending it for unbounded data, so we
have started adding support for things like triggers, watermarks and such
and we are wondering how come we can't find ValidatesRunner tests specific
to unbounded data. Tests from the UsesTestStream category seem to be kind
of a candidate for this, but they have nowhere near the coverage and
completeness provided by the ValidatesRunner ones.

I think we are missing something and I don't know what... Could you pls.
advise?

Rgds,
Jozsef

Re: Hazelcast Jet Runner - validation tests

[Kenneth Knowles <ke...@apache.org>]

Robert - that appears to be a test of state & timers, not triggers. Should
work for testing that the watermark at least advances. We do already have
similar java-based ValidatesRunner tests in ParDoTest.

The results of triggering, while nondeterministic, should generally fall
into a testable equivalence class, since the different outputs ought to
form a meaningful changelog of the evolution of the result. That is why
PAssert has methods like inCombinedNonLatePanes. For a triggered combine,
any test that examines the result more closely is overspecified. One
problem is that today the burden of ensuring that the results fall into a
well-defined equivalence class is split between the runner and the user.
The user needs to know about the trigger and accumulation mode and adjust
for it. This is not a blocker: the user in this case is the test, so it can
set things up appropriately, and that is how that PAssert method and others
work.

But none of that allows you to test that triggers are actually really
affecting the result. Partly this is because a runner is not *required* to
actually produce early output. The trigger *allows* the runner to fire, but
does not require it. It is not practical, nor the purpose of triggers, to
require that exact particular elements be produced. Our runners, and their
streaming vs batch mode, have very different behavior in this regard, and
it is generally coupled to bundling where they also differ.

All that said, the testing situation could be better. IMO implementing
TestStream would be the best way, but requires implementing knowledge of
global quiescence (or at least a reliable-enough approximation). This
feature would, itself, be useful for users debugging stuckness...

If you use ReduceFnRunner to execute triggers then it has a lot of testing,
so you just need to make sure your integration is basically right. There's
a Jira to convert these to TestStream-based tests - they were written
before TestStream existed.

Kenn

On Fri, Apr 5, 2019 at 1:37 AM Robert Bradshaw <ro...@google.com> wrote:

> On Thu, Apr 4, 2019 at 6:38 PM Lukasz Cwik <lc...@google.com> wrote:
> >
> > The issue with unbounded tests that rely on triggers/late data/early
> firings/processing time is that these are several sources of
> non-determinism. The sources make non-deterministic decisions around when
> to produce data, checkpoint, and resume and runners make non-deterministic
> decisions around when to output elements, in which order, and when to
> evaluate triggers. UsesTestStream is the best set of tests we currently
> have for making non-deterministic processing decisions deterministic but
> are more difficult to write then the other ValidatesRunner tests and also
> not well supported because of the special nature of UsesTestStream needing
> special hooks within the runner to control when to output and when to
> advance time.
> >
> > I'm not aware of any tests that we currently have that run a non
> deterministic pipeline and evaluate it against all possible outcomes that
> could have been produced and check that the output was valid. We would
> welcome ideas in how to improve this space to get more runners being tested
> for non-deterministic pipelines.
>
> Python has some tests of this nature, e.g.
>
>
> https://github.com/apache/beam/blob/release-2.12.0/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py#L308
>
> I'd imagine we could do similar for Java.
>
> > On Thu, Apr 4, 2019 at 3:36 AM Jozsef Bartok <jo...@hazelcast.com>
> wrote:
> >>
> >> Hi.
> >>
> >> My name is Jozsef, I've been working on Runners based on Hazelcast Jet.
> Plural because we have both an "old-style" and a "portable" Runner in
> development (https://github.com/hazelcast/hazelcast-jet-beam-runner).
> >>
> >> While our portable one isn't even functional yet, the "old-style" type
> of Runner is a bit more mature. It handles only bounded data, but for that
> case it passes all Beam tests of ValidatesRunner category and runs the
> Nexmark suite successfully too (I'm refering only to correctness, because
> performance is not yet where it can be, we aren't doing any Pipeline
> surgery yet and no other optimizations either).
> >>
> >> Since a few days we have started extending it for unbounded data, so we
> have started adding support for things like triggers, watermarks and such
> and we are wondering how come we can't find ValidatesRunner tests specific
> to unbounded data. Tests from the UsesTestStream category seem to be kind
> of a candidate for this, but they have nowhere near the coverage and
> completeness provided by the ValidatesRunner ones.
> >>
> >> I think we are missing something and I don't know what... Could you
> pls. advise?
> >>
> >> Rgds,
> >> Jozsef
>

Re: Hazelcast Jet Runner - validation tests

[Robert Bradshaw <ro...@google.com>]

On Thu, Apr 4, 2019 at 6:38 PM Lukasz Cwik <lc...@google.com> wrote:
>
> The issue with unbounded tests that rely on triggers/late data/early firings/processing time is that these are several sources of non-determinism. The sources make non-deterministic decisions around when to produce data, checkpoint, and resume and runners make non-deterministic decisions around when to output elements, in which order, and when to evaluate triggers. UsesTestStream is the best set of tests we currently have for making non-deterministic processing decisions deterministic but are more difficult to write then the other ValidatesRunner tests and also not well supported because of the special nature of UsesTestStream needing special hooks within the runner to control when to output and when to advance time.
>
> I'm not aware of any tests that we currently have that run a non deterministic pipeline and evaluate it against all possible outcomes that could have been produced and check that the output was valid. We would welcome ideas in how to improve this space to get more runners being tested for non-deterministic pipelines.

Python has some tests of this nature, e.g.

https://github.com/apache/beam/blob/release-2.12.0/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py#L308

I'd imagine we could do similar for Java.

> On Thu, Apr 4, 2019 at 3:36 AM Jozsef Bartok <jo...@hazelcast.com> wrote:
>>
>> Hi.
>>
>> My name is Jozsef, I've been working on Runners based on Hazelcast Jet. Plural because we have both an "old-style" and a "portable" Runner in development (https://github.com/hazelcast/hazelcast-jet-beam-runner).
>>
>> While our portable one isn't even functional yet, the "old-style" type of Runner is a bit more mature. It handles only bounded data, but for that case it passes all Beam tests of ValidatesRunner category and runs the Nexmark suite successfully too (I'm refering only to correctness, because performance is not yet where it can be, we aren't doing any Pipeline surgery yet and no other optimizations either).
>>
>> Since a few days we have started extending it for unbounded data, so we have started adding support for things like triggers, watermarks and such and we are wondering how come we can't find ValidatesRunner tests specific to unbounded data. Tests from the UsesTestStream category seem to be kind of a candidate for this, but they have nowhere near the coverage and completeness provided by the ValidatesRunner ones.
>>
>> I think we are missing something and I don't know what... Could you pls. advise?
>>
>> Rgds,
>> Jozsef

Re: Hazelcast Jet Runner - validation tests

[Lukasz Cwik <lc...@google.com>]

The issue with unbounded tests that rely on triggers/late data/early
firings/processing time is that these are several sources of
non-determinism. The sources make non-deterministic decisions around when
to produce data, checkpoint, and resume and runners make non-deterministic
decisions around when to output elements, in which order, and when to
evaluate triggers. UsesTestStream is the best set of tests we currently
have for making non-deterministic processing decisions deterministic but
are more difficult to write then the other ValidatesRunner tests and also
not well supported because of the special nature of UsesTestStream needing
special hooks within the runner to control when to output and when to
advance time.

I'm not aware of any tests that we currently have that run a non
deterministic pipeline and evaluate it against all possible outcomes that
could have been produced and check that the output was valid. We would
welcome ideas in how to improve this space to get more runners being tested
for non-deterministic pipelines.

On Thu, Apr 4, 2019 at 3:36 AM Jozsef Bartok <jo...@hazelcast.com> wrote:

> Hi.
>
> My name is Jozsef, I've been working on Runners based on Hazelcast Jet.
> Plural because we have both an "old-style" and a "portable" Runner in
> development (https://github.com/hazelcast/hazelcast-jet-beam-runner).
>
> While our portable one isn't even functional yet, the "old-style" type of
> Runner is a bit more mature. It handles only bounded data, but for that
> case it passes all Beam tests of ValidatesRunner category and runs the
> Nexmark suite successfully too (I'm refering only to correctness, because
> performance is not yet where it can be, we aren't doing any Pipeline
> surgery yet and no other optimizations either).
>
> Since a few days we have started extending it for unbounded data, so we
> have started adding support for things like triggers, watermarks and such
> and we are wondering how come we can't find ValidatesRunner tests specific
> to unbounded data. Tests from the UsesTestStream category seem to be kind
> of a candidate for this, but they have nowhere near the coverage and
> completeness provided by the ValidatesRunner ones.
>
> I think we are missing something and I don't know what... Could you pls.
> advise?
>
> Rgds,
> Jozsef
>