You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/07/21 00:39:00 UTC
[jira] [Commented] (BEAM-2512) TextIO should support watching for
new files
[ https://issues.apache.org/jira/browse/BEAM-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095601#comment-16095601 ]
ASF GitHub Bot commented on BEAM-2512:
--------------------------------------
GitHub user jkff opened a pull request:
https://github.com/apache/beam/pull/3607
[BEAM-2512] Introduces TextIO.read/readAll().watchForNewFiles()
https://issues.apache.org/jira/browse/BEAM-2512
Part of http://s.apache.org/textio-sdf, based on http://s.apache.org/beam-watch-transform.
This PR includes https://github.com/apache/beam/pull/3565 - reviewer should look only at the other commit. Also, requires https://github.com/apache/beam/pull/3598 to properly support read().from(ValueProvider).watchForNewFiles() - this PR should be submitted only after both of the PRs above.
R: @reuvenlax
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam textio-read-watch-new-files
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3607.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3607
----
commit 3103f9438a9fc392dfa1c37ceac990fc43c2ab98
Author: Eugene Kirpichov <ki...@google.com>
Date: 2017-07-20T02:50:03Z
[BEAM-2623] Introduces Watch transform
The transform watches for new elements in a family of growing sets.
See design at http://s.apache.org/beam-watch-transform
As part of the implementation, I found and fixed a bug in tracking the
watermark in OutputAndTimeBoundedSplittableProcessElementInvoker.
The watermark must be captured at the moment checkpoint is taken,
because it describes timestamps of elements output from the checkpoint.
I also made direct runner by default checkpoint SDF's every 100 elements
rather than every 10000, to make it more aggressive - that's what
uncovered the bug above.
commit c977d606f14a59ed73acf22f32a6b250d89c0ccd
Author: Eugene Kirpichov <ki...@google.com>
Date: 2017-07-20T23:58:42Z
[BEAM-2512] Introduces TextIO.read/readAll().watchForNewFiles()
Part of http://s.apache.org/textio-sdf, based on
http://s.apache.org/beam-watch-transform.
----
> TextIO should support watching for new files
> --------------------------------------------
>
> Key: BEAM-2512
> URL: https://issues.apache.org/jira/browse/BEAM-2512
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
> Reporter: Eugene Kirpichov
> Assignee: Eugene Kirpichov
>
> Motivation and proposed implementation in https://s.apache.org/textio-sdf
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)