You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by jkff <gi...@git.apache.org> on 2017/07/21 00:38:17 UTC
[GitHub] beam pull request #3607: [BEAM-2512] Introduces TextIO.read/readAll().watchF...
GitHub user jkff opened a pull request:
https://github.com/apache/beam/pull/3607
[BEAM-2512] Introduces TextIO.read/readAll().watchForNewFiles()
https://issues.apache.org/jira/browse/BEAM-2512
Part of http://s.apache.org/textio-sdf, based on http://s.apache.org/beam-watch-transform.
This PR includes https://github.com/apache/beam/pull/3565 - reviewer should look only at the other commit. Also, requires https://github.com/apache/beam/pull/3598 to properly support read().from(ValueProvider).watchForNewFiles() - this PR should be submitted only after both of the PRs above.
R: @reuvenlax
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam textio-read-watch-new-files
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3607.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3607
----
commit 3103f9438a9fc392dfa1c37ceac990fc43c2ab98
Author: Eugene Kirpichov <ki...@google.com>
Date: 2017-07-20T02:50:03Z
[BEAM-2623] Introduces Watch transform
The transform watches for new elements in a family of growing sets.
See design at http://s.apache.org/beam-watch-transform
As part of the implementation, I found and fixed a bug in tracking the
watermark in OutputAndTimeBoundedSplittableProcessElementInvoker.
The watermark must be captured at the moment checkpoint is taken,
because it describes timestamps of elements output from the checkpoint.
I also made direct runner by default checkpoint SDF's every 100 elements
rather than every 10000, to make it more aggressive - that's what
uncovered the bug above.
commit c977d606f14a59ed73acf22f32a6b250d89c0ccd
Author: Eugene Kirpichov <ki...@google.com>
Date: 2017-07-20T23:58:42Z
[BEAM-2512] Introduces TextIO.read/readAll().watchForNewFiles()
Part of http://s.apache.org/textio-sdf, based on
http://s.apache.org/beam-watch-transform.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] beam pull request #3607: [BEAM-2512] Introduces TextIO.watchForNewFiles() an...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/beam/pull/3607
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---