You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/03/31 15:50:25 UTC

[jira] [Commented] (BEAM-158) Add support for bounded sources in streaming mode

    [ https://issues.apache.org/jira/browse/BEAM-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219884#comment-15219884 ] 

ASF GitHub Bot commented on BEAM-158:
-------------------------------------

GitHub user mxm opened a pull request:

    https://github.com/apache/incubator-beam/pull/104

    [BEAM-158] add support for bounded sources in streaming

    Apart from a few improvements, this PR introduces bounded sources in streaming. The BoundedSource wrapper (`SourceInputFormat`) is the same as for the batch part of the runner. The translator assigns ingestion time watermarks and processing time timestamps upon reading from the source. We could make this more flexible in terms of watermark generation if we had an UnboundedSource wrapper for a BoundedSource.
    
    Perhaps we could have common utility for runners to deal with serialization of PipelineOptions. At some point, they have to be shipped. I had to change the serialization code because I was experiencing a serialization bug which led to a serialization loop. Debugging this was almost impossible because the stack trace doesn't show all serialization calls due to some magic in the VM. I didn't find any cyclic references between the PipelineOptions and Flink components. I'm assuming this is a bug and the workaround using byte array serialization of the options is fair enough. See `SourceInputFormat`.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mxm/incubator-beam BEAM-158

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/104.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #104
    
----
commit f26674e7a7b30ee3f992edfc8e473df2a7ee3e80
Author: Maximilian Michels <mx...@apache.org>
Date:   2016-03-30T14:43:04Z

    [flink] improve InputFormat wrapper and ReadSourceITCase

commit 03404c7f4a5656bb5c5c0a2510f12e33292fef01
Author: Maximilian Michels <mx...@apache.org>
Date:   2016-03-30T17:05:27Z

    [flink] improvements to UnboundedSource translation

commit de574136a5b4ba9b75231b321d0190e23af3bac2
Author: Maximilian Michels <mx...@apache.org>
Date:   2016-03-31T08:18:01Z

    [BEAM-158] add support for bounded sources in streaming

----


> Add support for bounded sources in streaming mode
> -------------------------------------------------
>
>                 Key: BEAM-158
>                 URL: https://issues.apache.org/jira/browse/BEAM-158
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-flink
>            Reporter: Maximilian Michels
>            Assignee: Maximilian Michels
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)