You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "johannaojeling (via GitHub)" <gi...@apache.org> on 2023/04/09 12:52:00 UTC

[GitHub] [beam] johannaojeling opened a new pull request, #26188: [Go SDK]: Implement fileio.MatchContinuously transform

johannaojeling opened a new pull request, #26188:
URL: https://github.com/apache/beam/pull/26188

   Fixes #26186 and adds a `fileio.MatchContinuously` transform to the Go SDK.
   
   I have tested the transform manually on Dataflow and Flink, which seem to be the only runners that currently support all of watermark estimation, self-checkpointing and state provisioning, that this transform relies on. I have not added any unit tests for the same reason, and there were limitations with running integration tests with some of these features for Dataflow and Flink so I have skipped that as well.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #26188:
URL: https://github.com/apache/beam/pull/26188#issuecomment-1511471132

   R: @lostluck 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26188:
URL: https://github.com/apache/beam/pull/26188#issuecomment-1501130349

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @riteshghorse for label go.
   R: @pabloem for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] lostluck commented on pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #26188:
URL: https://github.com/apache/beam/pull/26188#issuecomment-1515487535

   Happy to merge this in if there are no further question/disccusion (after one of us fixes the conflict in CHANGES.md of course!)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] lostluck merged pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck merged PR #26188:
URL: https://github.com/apache/beam/pull/26188


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] lostluck commented on a diff in pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #26188:
URL: https://github.com/apache/beam/pull/26188#discussion_r1171919456


##########
sdks/go/pkg/beam/io/fileio/match.go:
##########
@@ -187,3 +198,146 @@ func metadataFromFiles(
 
 	return metadata, nil
 }
+
+// duplicateTreatment controls how duplicate matches are treated.
+type duplicateTreatment int
+
+const (
+	// duplicateAllow allows duplicate matches.
+	duplicateAllow duplicateTreatment = iota
+	// duplicateSkip skips duplicate matches.
+	duplicateSkip
+)
+
+type matchContOption struct {
+	Start              time.Time
+	End                time.Time
+	DuplicateTreatment duplicateTreatment
+	ApplyWindow        bool
+}
+
+// MatchContOptionFn is a function that can be passed to MatchContinuously to configure options for
+// matching files.
+type MatchContOptionFn func(*matchContOption)
+
+// MatchStart specifies the start time for matching files.
+func MatchStart(start time.Time) MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.Start = start
+	}
+}
+
+// MatchEnd specifies the end time for matching files.
+func MatchEnd(end time.Time) MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.End = end
+	}
+}
+
+// MatchDuplicateAllow specifies that file path matches will not be deduplicated.
+func MatchDuplicateAllow() MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.DuplicateTreatment = duplicateAllow
+	}
+}
+
+// MatchDuplicateSkip specifies that file path matches will be deduplicated.
+func MatchDuplicateSkip() MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.DuplicateTreatment = duplicateSkip
+	}
+}
+
+// MatchApplyWindow specifies that each element will be assigned to an individual window.
+func MatchApplyWindow() MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.ApplyWindow = true
+	}
+}
+
+// MatchContinuously finds all files matching the glob pattern at the given interval and returns a
+// PCollection<FileMetadata> of the matching files. MatchContinuously accepts a variadic number of
+// MatchContOptionFn that can be used to configure:
+//
+//   - Start: start time for matching files. Defaults to the current timestamp
+//   - End: end time for matching files. Defaults to the maximum timestamp
+//   - DuplicateAllow: allow emitting matches that have already been observed. Defaults to false
+//   - DuplicateSkip: skip emitting matches that have already been observed. Defaults to true

Review Comment:
   I'll review PR 26192 first then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26188:
URL: https://github.com/apache/beam/pull/26188#issuecomment-1511473904

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26188:
URL: https://github.com/apache/beam/pull/26188#issuecomment-1511228007

   Reminder, please take a look at this pr: @riteshghorse @pabloem 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26188:
URL: https://github.com/apache/beam/pull/26188#issuecomment-1501124184

   ## [Codecov](https://codecov.io/gh/apache/beam/pull/26188?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#26188](https://codecov.io/gh/apache/beam/pull/26188?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (81bd33f) into [master](https://codecov.io/gh/apache/beam/commit/dc7f9d11498b5bec567bcb5f29ff27ed2c1fde23?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dc7f9d1) will **not change** coverage.
   > The diff coverage is `n/a`.
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #26188   +/-   ##
   =======================================
     Coverage   71.18%   71.18%           
   =======================================
     Files         787      787           
     Lines      103293   103293           
   =======================================
     Hits        73525    73525           
     Misses      28286    28286           
     Partials     1482     1482           
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/26188?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/go/pkg/beam/io/fileio/match.go](https://codecov.io/gh/apache/beam/pull/26188?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlaW8vbWF0Y2guZ28=) | `81.39% <ø> (ø)` | |
   | [sdks/go/pkg/beam/io/fileio/read.go](https://codecov.io/gh/apache/beam/pull/26188?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlaW8vcmVhZC5nbw==) | `79.31% <ø> (ø)` | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] johannaojeling commented on pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "johannaojeling (via GitHub)" <gi...@apache.org>.
johannaojeling commented on PR #26188:
URL: https://github.com/apache/beam/pull/26188#issuecomment-1515733717

   Thank you both for reviewing!
   
   The Java and Python equivalents don't use timers. Happy too add tests for this one later on when state is supported.
   
   I can open a separate PR for the modified files part once #26192 has been merged so please go ahead and merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] johannaojeling commented on a diff in pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "johannaojeling (via GitHub)" <gi...@apache.org>.
johannaojeling commented on code in PR #26188:
URL: https://github.com/apache/beam/pull/26188#discussion_r1161282399


##########
sdks/go/pkg/beam/io/fileio/match.go:
##########
@@ -187,3 +198,146 @@ func metadataFromFiles(
 
 	return metadata, nil
 }
+
+// duplicateTreatment controls how duplicate matches are treated.
+type duplicateTreatment int
+
+const (
+	// duplicateAllow allows duplicate matches.
+	duplicateAllow duplicateTreatment = iota
+	// duplicateSkip skips duplicate matches.
+	duplicateSkip
+)
+
+type matchContOption struct {
+	Start              time.Time
+	End                time.Time
+	DuplicateTreatment duplicateTreatment
+	ApplyWindow        bool
+}
+
+// MatchContOptionFn is a function that can be passed to MatchContinuously to configure options for
+// matching files.
+type MatchContOptionFn func(*matchContOption)
+
+// MatchStart specifies the start time for matching files.
+func MatchStart(start time.Time) MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.Start = start
+	}
+}
+
+// MatchEnd specifies the end time for matching files.
+func MatchEnd(end time.Time) MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.End = end
+	}
+}
+
+// MatchDuplicateAllow specifies that file path matches will not be deduplicated.
+func MatchDuplicateAllow() MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.DuplicateTreatment = duplicateAllow
+	}
+}
+
+// MatchDuplicateSkip specifies that file path matches will be deduplicated.
+func MatchDuplicateSkip() MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.DuplicateTreatment = duplicateSkip
+	}
+}
+
+// MatchApplyWindow specifies that each element will be assigned to an individual window.
+func MatchApplyWindow() MatchContOptionFn {
+	return func(o *matchContOption) {
+		o.ApplyWindow = true
+	}
+}
+
+// MatchContinuously finds all files matching the glob pattern at the given interval and returns a
+// PCollection<FileMetadata> of the matching files. MatchContinuously accepts a variadic number of
+// MatchContOptionFn that can be used to configure:
+//
+//   - Start: start time for matching files. Defaults to the current timestamp
+//   - End: end time for matching files. Defaults to the maximum timestamp
+//   - DuplicateAllow: allow emitting matches that have already been observed. Defaults to false
+//   - DuplicateSkip: skip emitting matches that have already been observed. Defaults to true

Review Comment:
   I would like to also enable emitting a file only if it has been modified since the last observation, but it requires introduction of a new method to the filesystems for retrieving a file's last modified timestamp first (#26187) + attaching this info to the `fileio.FileMetadata` in `fileio.matchFn`, so I will do it at a later stage



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] lostluck commented on pull request #26188: [Go SDK]: Implement fileio.MatchContinuously transform

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #26188:
URL: https://github.com/apache/beam/pull/26188#issuecomment-1515473419

   Ah foiled! We can fix this a bit once state is added to Prism, since that's what's currently missing here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org