You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Eliaaazzz (via GitHub)" <gi...@apache.org> on 2026/07/16 14:16:01 UTC
[PR] [Python] Add Watch transform for MatchContinuously [beam]
Eliaaazzz opened a new pull request, #39352:
URL: https://github.com/apache/beam/pull/39352
This PR adds an experimental Python `Watch` transform and refactors the deduplicated `FileIO.MatchContinuously` path to use it.
`Watch` polls each input with a user-provided `PollFn`, keeps only never-seen-before outputs, advances output watermarks, and supports per-input termination. The default deduplication behavior is by output key. A new `use_timestamp` option lets callers include the output event timestamp in the dedup key.
`MatchContinuously` now uses `Watch` when `has_deduplication=True`. Normal deduplication is still by file path. When `match_updated_files=True`, the poll function timestamps each `FileMetadata` output with `last_updated_in_seconds`, and `Watch(use_timestamp=True)` emits the same path again when that timestamp changes. The `has_deduplication=False` path remains `PeriodicImpulse` plus `MatchAll` so every poll can re-emit all matches.
Validation:
- `python -m pytest -q sdks/python/apache_beam/io/watch_test.py sdks/python/apache_beam/io/fileio_test.py::MatchContinuouslyTest`
- `git diff --check upstream/master..HEAD`
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description, if applicable. No issue is currently linked.
- [ ] Update `CHANGES.md` with noteworthy changes. Not updated; this is an experimental Python SDK transform/refactor.
- [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Refactor MatchContinuously to use Watch [beam]
Posted by "Eliaaazzz (via GitHub)" <gi...@apache.org>.
Eliaaazzz commented on PR #39352:
URL: https://github.com/apache/beam/pull/39352#issuecomment-5092726190
Superseded by #39461 (same refactor, reworked per review on the merged Watch). Closing to keep the review in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Refactor MatchContinuously to use Watch [beam]
Posted by "Eliaaazzz (via GitHub)" <gi...@apache.org>.
Eliaaazzz closed pull request #39352: [Python] Refactor MatchContinuously to use Watch
URL: https://github.com/apache/beam/pull/39352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org