You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "johannaojeling (via GitHub)" <gi...@apache.org> on 2023/05/03 15:59:36 UTC
[GitHub] [beam] johannaojeling opened a new pull request, #26524: [Go SDK]: Enable fileio.MatchContinuously to emit duplicate file if modified
johannaojeling opened a new pull request, #26524:
URL: https://github.com/apache/beam/pull/26524
Updates the `fileio.MatchContinuously` transform to support emission of duplicate matches if a file has been modified since it was last observed. The behavior is enabled by passing the `fileio.MatchDuplicateAllowIfModified()` option. Fixes #26523.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #26524: [Go SDK]: Enable fileio.MatchContinuously to emit duplicate file if modified
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #26524:
URL: https://github.com/apache/beam/pull/26524#discussion_r1194395867
##########
sdks/go/pkg/beam/io/fileio/match.go:
##########
@@ -190,21 +194,45 @@ func metadataFromFiles(
return nil, err
}
+ mTime, err := lastModified(ctx, fs, path)
+ if err != nil {
+ return nil, err
+ }
+
metadata[i] = FileMetadata{
- Path: path,
- Size: size,
+ Path: path,
+ Size: size,
+ LastModified: mTime,
}
}
return metadata, nil
}
+func lastModified(ctx context.Context, fs filesystem.Interface, path string) (time.Time, error) {
Review Comment:
Not in this PR, but we could consider putting this as a helper function in the filesystem package instead, to allow more reasonable re-use and consistency in the implementation:
Eg. We implement copy, but if there's a Copier implementation, it can be used instead.
https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/io/filesystem/util.go#L55
We'd probably want a "creation time" too though when we do that, so the default isn't the zero time. Not critical at this stage though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johannaojeling commented on pull request #26524: [Go SDK]: Enable fileio.MatchContinuously to emit duplicate file if modified
Posted by "johannaojeling (via GitHub)" <gi...@apache.org>.
johannaojeling commented on PR #26524:
URL: https://github.com/apache/beam/pull/26524#issuecomment-1533303777
R: @lostluck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #26524: [Go SDK]: Enable fileio.MatchContinuously to emit duplicate file if modified
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #26524:
URL: https://github.com/apache/beam/pull/26524#issuecomment-1544921710
Thanks for your patience. I was on vacation towards the end of last week, and just caught up on things. Looking at this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26524: [Go SDK]: Enable fileio.MatchContinuously to emit duplicate file if modified
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26524:
URL: https://github.com/apache/beam/pull/26524#issuecomment-1533305814
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #26524: [Go SDK]: Enable fileio.MatchContinuously to emit duplicate file if modified
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck merged PR #26524:
URL: https://github.com/apache/beam/pull/26524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #26524: [Go SDK]: Enable fileio.MatchContinuously to emit duplicate file if modified
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #26524:
URL: https://github.com/apache/beam/pull/26524#issuecomment-1548706778
Oh dang, I will send a quick fix for this, but it looks like the time range might be incorrectly set for the local unit test:
Caught in another CL.
```
-- FAIL: TestLocal_lastModified (0.00s)
local_test.go:177: LastModified("/tmp/TestLocal_lastModified2985575980/001/file.txt") = 2023-05-15 21:32:42.998867243 +0000 UTC, want in range [2023-05-15 21:32:43 +0000 UTC, 2023-05-15 21:32:43.004302634 +0000 UTC m=+0.001527498]
FAIL
github.com/apache/beam/sdks/v2/go/pkg/beam/io/filesystem/local coverage: 77.4% of statements
FAIL github.com/apache/beam/sdks/v2/go/pkg/beam/io/filesystem/local 0.036s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org