[jira] [Commented] (BEAM-604) Use Watermark Check Streaming Job Finish in DataflowPipelineJob

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15459547#comment-15459547 ] 

ASF GitHub Bot commented on BEAM-604:
-------------------------------------

GitHub user markflyhigh opened a pull request:

    https://github.com/apache/incubator-beam/pull/916

    [BEAM-604] Use Watermark to Finish Streaming Job in TestDataflowRunner

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [x] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
    
    ---
    
     - Add checkMaxWatermark() function in TestDataflowRunner, so that when testing on streaming pipeline with bounded input, the job can be canceled as soon as all watermark reach to max value (by default is -2). Then, verification steps can be executed.
     - Add WindowedWordCountIT as a basic example of testing on streaming job.
     - Add non-terminated check before canceling steaming job.
    
    TODO:
     - Create verifier for WindowedWordCountIT. (BigQuery verifier)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/markflyhigh/incubator-beam streaming-wait-until-max-watermark

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/916.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #916
    
----
commit bb8a6f8a360a8f263fe0ad625ab4159d645c42ab
Author: Mark Liu <ma...@markliu-macbookpro.roam.corp.google.com>
Date:   2016-09-02T20:22:37Z

    [BEAM-604] Use Watermark to Finish Streaming Job in TestDataflowRunner

----


> Use Watermark Check Streaming Job Finish in DataflowPipelineJob
> ---------------------------------------------------------------
>
>                 Key: BEAM-604
>                 URL: https://issues.apache.org/jira/browse/BEAM-604
>             Project: Beam
>          Issue Type: Improvement
>            Reporter: Mark Liu
>            Assignee: Mark Liu
>            Priority: Minor
>
> Currently, streaming job with bounded input can't be terminated automatically and TestDataflowRunner can't handle this case. Need to update TestDataflowRunner so that streaming integration test such as WindowedWordCountIT can run with it.
> Implementation:
> Query watermark of each step and wait until all watermarks set to MAX then cancel the job.
> Update:
> Suggesting by [~peihe0@gmail.com], implement checkMaxWatermark in DataflowPipelineJob#waitUntilFinish. Thus, all dataflow streaming jobs with bounded input will take advantage of this change and are canceled automatically when watermarks reach to max value. Also Dataflow runners can keep simple and free from handling batch and streaming two cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)