[jira] [Commented] (BEAM-735) PAssertStreaming should make sure the assertion happened.

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560432#comment-15560432 ] 

ASF GitHub Bot commented on BEAM-735:
-------------------------------------

GitHub user amitsela opened a pull request:

    https://github.com/apache/incubator-beam/pull/1073

    [BEAM-735] PAssertStreaming should make sure the assertion happened.

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
    
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/amitsela/incubator-beam BEAM-735

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/1073.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1073
    
----
commit ca3aa78d9651d8d8fc981234c9f4707414dc9738
Author: Sela <an...@paypal.com>
Date:   2016-10-09T18:38:14Z

    PAssertStreaming shuold check an assertion happened.

commit 4adc3827443a6d8490f6b1799fce5c820c9484a5
Author: Sela <an...@paypal.com>
Date:   2016-10-09T18:39:00Z

    Test assert for skipped assertion..

commit 948fc9d0e894c06e555cf47fc3db48e6aad55008
Author: Sela <an...@paypal.com>
Date:   2016-10-09T18:39:53Z

    This name is more true to the natureof this test.

commit 346e85df4436a07a8dd30d793d7fa0ca4bf23806
Author: Sela <an...@paypal.com>
Date:   2016-10-09T18:40:46Z

    Fix according to new PAssertStreaming.

----


> PAssertStreaming should make sure the assertion happened.
> ---------------------------------------------------------
>
>                 Key: BEAM-735
>                 URL: https://issues.apache.org/jira/browse/BEAM-735
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Amit Sela
>
> The Spark runner currently runs PAsserts via `PAssertStreaming` which groups into a single key and asserts the values on the worker (part of the "Lambda" in the Spark lingo).
> This could be a problem since Spark won't run if there's nothing to process - so that if for some reason the input is missed, say reading from Kafka latest or simply an empty topic, the assertion will be skipped and so we'll never fail (we would like to fail if there was no input, while we expected one).
> This might change once Spark provide a better support for the Beam model in streaming, but until then, it's best that our tests will consider this case as well.
> I'll add an aggregator and increment for assertion, at the end I'll make sure the aggregator is not 0, so that at least one assertion took place (if for some reason Spark kept on for a couple of more intervals it might execute the same assertion more then once, if the input is repeated).  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)