You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "prodriguezdefino (via GitHub)" <gi...@apache.org> on 2023/11/29 23:19:36 UTC
[PR] Generalize file based source output capabilities [beam]
prodriguezdefino opened a new pull request, #29566:
URL: https://github.com/apache/beam/pull/29566
Currently, when using a file based source implementation to read data from files we have 2 options: read only the content of the each file into a PCollection or read the content of a file, key it with the file name and put it into a PCollection. While this cover a large bulk of the desired read operations, we can generalize the current implementation to allow a much larger space of possibilities for the users.
By introducing a serializable lambda as a parameter for the `ReadAllViaFileBasedSource` class, and a data carrier adapter to simplify the makeOutput method arguments with an arbitrarily provided lambda, we can enable users to make interesting decisions about how to treat the data read from particular files by making more information accessible about the files their data was extracted from.
This change adds such generalization to the TextIO PTransform and an example that extracts not only a file name with the data line contained in the file, but also the originally file size which can be used to predicate, for example, if the destination file should be partitioned in many given the input one size.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967336923
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1913138465
Reminder, please take a look at this pr: @Abacn @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1918989406
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damondouglas for label java.
R: @damondouglas for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1855739970
Reminder, please take a look at this pr: @bvolpato @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1878573575
Reminder, please take a look at this pr: @damondouglas @damondouglas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1893625576
Reminder, please take a look at this pr: @robertwb @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #29566: Generalize file based source output capabilities
URL: https://github.com/apache/beam/pull/29566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-2184973869
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1941382305
Reminder, please take a look at this pr: @damondouglas @damondouglas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1882963790
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @robertwb for label java.
R: @johnjcasey for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1841541894
Run Java_IOs_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967277975
Good day, @prodriguezdefino, Is this PR still active? Would you be able to investigate the two failing checks? Please let me know if you need anything and thank you again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on code in PR #29566:
URL: https://github.com/apache/beam/pull/29566#discussion_r1475338512
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
* <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
* FileIO#readMatches()}.
*/
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
- public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+ private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+ protected ReadAllViaFileBasedSource(
long desiredBundleSizeBytes,
SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- Coder<T> coder) {
+ Coder<K> coder,
+ SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
super(
desiredBundleSizeBytes,
createSource,
coder,
DEFAULT_USES_RESHUFFLE,
- new ReadFileRangesFnExceptionHandler());
+ new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+ this.outputFn = outputFn;
}
- public ReadAllViaFileBasedSource(
+ protected ReadAllViaFileBasedSource(
Review Comment:
Would the reduction in visibility break others code who may already depend on this?
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
* <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
* FileIO#readMatches()}.
*/
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
- public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+ private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+ protected ReadAllViaFileBasedSource(
long desiredBundleSizeBytes,
SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- Coder<T> coder) {
+ Coder<K> coder,
+ SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
super(
desiredBundleSizeBytes,
createSource,
coder,
DEFAULT_USES_RESHUFFLE,
- new ReadFileRangesFnExceptionHandler());
+ new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+ this.outputFn = outputFn;
}
- public ReadAllViaFileBasedSource(
+ protected ReadAllViaFileBasedSource(
long desiredBundleSizeBytes,
SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- Coder<T> coder,
+ Coder<K> coder,
boolean usesReshuffle,
- ReadFileRangesFnExceptionHandler exceptionHandler) {
+ ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler,
+ SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
super(desiredBundleSizeBytes, createSource, coder, usesReshuffle, exceptionHandler);
+ this.outputFn = outputFn;
+ }
+
+ public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+ long desiredBundleSizeBytes,
+ SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+ Coder<InputT> coder,
+ boolean usesReshuffle,
+ ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler) {
+ return new ReadAllViaFileBasedSource<>(
+ desiredBundleSizeBytes,
+ createSource,
+ coder,
+ usesReshuffle,
+ exceptionHandler,
+ outputArguments -> outputArguments.reader().getCurrent());
+ }
+
+ public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+ long desiredBundleSizeBytes,
+ SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+ Coder<InputT> coder) {
+ return create(
+ desiredBundleSizeBytes,
+ createSource,
+ coder,
+ outputArguments -> outputArguments.reader().getCurrent());
+ }
+
+ public static <InputT, OutputT> ReadAllViaFileBasedSource<InputT, OutputT> create(
+ long desiredBundleSizeBytes,
+ SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+ Coder<OutputT> coder,
+ SerializableFunction<OutputContextFromFile<InputT>, OutputT> outputFn) {
+ return new ReadAllViaFileBasedSource<>(desiredBundleSizeBytes, createSource, coder, outputFn);
}
@Override
- protected DoFn<KV<ReadableFile, OffsetRange>, T> readRangesFn() {
- return new ReadFileRangesFn<>(createSource, exceptionHandler);
+ protected DoFn<KV<ReadableFile, OffsetRange>, K> readRangesFn() {
+ return new ReadFileRangesFn<>(outputFn, createSource, exceptionHandler);
}
- private static class ReadFileRangesFn<T> extends AbstractReadFileRangesFn<T, T> {
+ private static class ReadFileRangesFn<T, K> extends AbstractReadFileRangesFn<T, K> {
+ private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
public ReadFileRangesFn(
+ final SerializableFunction<OutputContextFromFile<T>, K> outputFn,
final SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- final ReadFileRangesFnExceptionHandler exceptionHandler) {
+ final ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler
Review Comment:
Would changing this signature of a public method break others code that already depends on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1833268429
assign set of reviewers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1833171066
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967331829
Sorry on the delay, checking on this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-2169498263
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think thatβs incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1833270082
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @bvolpato for label java.
R: @ahmedabu98 for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1833217233
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1842085568
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on code in PR #29566:
URL: https://github.com/apache/beam/pull/29566#discussion_r1476626897
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
* <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
* FileIO#readMatches()}.
*/
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
- public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+ private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+ protected ReadAllViaFileBasedSource(
long desiredBundleSizeBytes,
SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- Coder<T> coder) {
+ Coder<K> coder,
+ SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
super(
desiredBundleSizeBytes,
createSource,
coder,
DEFAULT_USES_RESHUFFLE,
- new ReadFileRangesFnExceptionHandler());
+ new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+ this.outputFn = outputFn;
}
- public ReadAllViaFileBasedSource(
+ protected ReadAllViaFileBasedSource(
long desiredBundleSizeBytes,
SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- Coder<T> coder,
+ Coder<K> coder,
boolean usesReshuffle,
- ReadFileRangesFnExceptionHandler exceptionHandler) {
+ ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler,
+ SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
super(desiredBundleSizeBytes, createSource, coder, usesReshuffle, exceptionHandler);
+ this.outputFn = outputFn;
+ }
+
+ public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+ long desiredBundleSizeBytes,
+ SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+ Coder<InputT> coder,
+ boolean usesReshuffle,
+ ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler) {
+ return new ReadAllViaFileBasedSource<>(
+ desiredBundleSizeBytes,
+ createSource,
+ coder,
+ usesReshuffle,
+ exceptionHandler,
+ outputArguments -> outputArguments.reader().getCurrent());
+ }
+
+ public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+ long desiredBundleSizeBytes,
+ SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+ Coder<InputT> coder) {
+ return create(
+ desiredBundleSizeBytes,
+ createSource,
+ coder,
+ outputArguments -> outputArguments.reader().getCurrent());
+ }
+
+ public static <InputT, OutputT> ReadAllViaFileBasedSource<InputT, OutputT> create(
+ long desiredBundleSizeBytes,
+ SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+ Coder<OutputT> coder,
+ SerializableFunction<OutputContextFromFile<InputT>, OutputT> outputFn) {
+ return new ReadAllViaFileBasedSource<>(desiredBundleSizeBytes, createSource, coder, outputFn);
}
@Override
- protected DoFn<KV<ReadableFile, OffsetRange>, T> readRangesFn() {
- return new ReadFileRangesFn<>(createSource, exceptionHandler);
+ protected DoFn<KV<ReadableFile, OffsetRange>, K> readRangesFn() {
+ return new ReadFileRangesFn<>(outputFn, createSource, exceptionHandler);
}
- private static class ReadFileRangesFn<T> extends AbstractReadFileRangesFn<T, T> {
+ private static class ReadFileRangesFn<T, K> extends AbstractReadFileRangesFn<T, K> {
+ private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
public ReadFileRangesFn(
+ final SerializableFunction<OutputContextFromFile<T>, K> outputFn,
final SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- final ReadFileRangesFnExceptionHandler exceptionHandler) {
+ final ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler
Review Comment:
This change obey the movement of that static class (`ReadFileRangesFnExceptionHandler`) to the parent class in the hierarchy. In master the code has a weird situation where a parent class depends on a static class implemented in one of the child classes.
I moved `ReadFileRangesFnExceptionHandler` to the top of the hierarchy to avoid the double reference, and changed the references in the Beam framework. Again this may break user code if they make direct reference to this static class, so I'm happy to revert that change if preferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1964590702
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "RCQE-bot (via GitHub)" <gi...@apache.org>.
RCQE-bot commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-2058847972
## RCQE-bot: Commit Quality Assessment π
I've carefully evaluated the recent commits, and here's the current state of the commit quality:
### Metrics π
- Previous Commits: diff hunk in file [sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java](https://github.com/apache/beam/pull/29566/files/349afba29fde04b3761f0d2956a84ed465355ef2#diff-8cecdc624834472e9bbcc9c2f5a129224d36d9f84a8e3316fe774d232af400e8) in commit [7e1eb23b904059c23dcd3d6fb13c39ce13a27930](https://github.com/apache/beam/pull/29566/commits/7e1eb23b904059c23dcd3d6fb13c39ce13a27930)
- Code Review: Would changing this signature of a public method break others code that already depends on this?
### Suggestions π‘
- Need Review Again π« : The commits do not currently meet the quality standards. I recommend not merging them into the main branch until the above suggestions have been implemented and a subsequent review confirms the quality. Please address the points mentioned and submit the commits for another round of review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1860329723
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @kennknowles for label java.
R: @johnjcasey for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1869499056
Reminder, please take a look at this pr: @kennknowles @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1843277476
This is a really interesting enhancement, but it goes to the core of a Beam transform. Do you have a design document for this? I want to make sure the community sees the trade offs here before we go too far into implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1964587970
This sounds like a good change.
Formally adding R: @damondouglas for follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on code in PR #29566:
URL: https://github.com/apache/beam/pull/29566#discussion_r1504737909
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
* <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
* FileIO#readMatches()}.
*/
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
- public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+ private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+ protected ReadAllViaFileBasedSource(
long desiredBundleSizeBytes,
SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- Coder<T> coder) {
+ Coder<K> coder,
+ SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
super(
desiredBundleSizeBytes,
createSource,
coder,
DEFAULT_USES_RESHUFFLE,
- new ReadFileRangesFnExceptionHandler());
+ new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+ this.outputFn = outputFn;
}
- public ReadAllViaFileBasedSource(
+ protected ReadAllViaFileBasedSource(
long desiredBundleSizeBytes,
SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- Coder<T> coder,
+ Coder<K> coder,
boolean usesReshuffle,
- ReadFileRangesFnExceptionHandler exceptionHandler) {
+ ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler,
+ SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
super(desiredBundleSizeBytes, createSource, coder, usesReshuffle, exceptionHandler);
+ this.outputFn = outputFn;
+ }
+
+ public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+ long desiredBundleSizeBytes,
+ SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+ Coder<InputT> coder,
+ boolean usesReshuffle,
+ ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler) {
+ return new ReadAllViaFileBasedSource<>(
+ desiredBundleSizeBytes,
+ createSource,
+ coder,
+ usesReshuffle,
+ exceptionHandler,
+ outputArguments -> outputArguments.reader().getCurrent());
+ }
+
+ public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+ long desiredBundleSizeBytes,
+ SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+ Coder<InputT> coder) {
+ return create(
+ desiredBundleSizeBytes,
+ createSource,
+ coder,
+ outputArguments -> outputArguments.reader().getCurrent());
+ }
+
+ public static <InputT, OutputT> ReadAllViaFileBasedSource<InputT, OutputT> create(
+ long desiredBundleSizeBytes,
+ SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+ Coder<OutputT> coder,
+ SerializableFunction<OutputContextFromFile<InputT>, OutputT> outputFn) {
+ return new ReadAllViaFileBasedSource<>(desiredBundleSizeBytes, createSource, coder, outputFn);
}
@Override
- protected DoFn<KV<ReadableFile, OffsetRange>, T> readRangesFn() {
- return new ReadFileRangesFn<>(createSource, exceptionHandler);
+ protected DoFn<KV<ReadableFile, OffsetRange>, K> readRangesFn() {
+ return new ReadFileRangesFn<>(outputFn, createSource, exceptionHandler);
}
- private static class ReadFileRangesFn<T> extends AbstractReadFileRangesFn<T, T> {
+ private static class ReadFileRangesFn<T, K> extends AbstractReadFileRangesFn<T, K> {
+ private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
public ReadFileRangesFn(
+ final SerializableFunction<OutputContextFromFile<T>, K> outputFn,
final SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- final ReadFileRangesFnExceptionHandler exceptionHandler) {
+ final ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler
Review Comment:
@damondouglas let me know if you want me to revert the movement of `ReadFileRangesFnExceptionHandler` to the top of the hierarchy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967337231
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967557849
Run Java_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1962345087
Reminder, please take a look at this pr: @robertwb @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1871110685
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damondouglas for label java.
R: @damondouglas for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1841972001
fixes #29627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1841697835
pull_request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on code in PR #29566:
URL: https://github.com/apache/beam/pull/29566#discussion_r1476620928
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
* <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
* FileIO#readMatches()}.
*/
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
- public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+ private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+ protected ReadAllViaFileBasedSource(
long desiredBundleSizeBytes,
SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
- Coder<T> coder) {
+ Coder<K> coder,
+ SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
super(
desiredBundleSizeBytes,
createSource,
coder,
DEFAULT_USES_RESHUFFLE,
- new ReadFileRangesFnExceptionHandler());
+ new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+ this.outputFn = outputFn;
}
- public ReadAllViaFileBasedSource(
+ protected ReadAllViaFileBasedSource(
Review Comment:
I'm happy to revert the visibility change on the constructor signature.
The other transforms that uses this class in Beam only accessed the functionality through inheritance. I understand the potential breakage of other user based transforms that may be impacted by this code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1948280091
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @robertwb for label java.
R: @ahmedabu98 for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1919753414
Good day, @prodriguezdefino and thank you for contributing! I will be taking a look at this and placed it in my review queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generalize file based source output capabilities [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1900308337
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @Abacn for label java.
R: @ahmedabu98 for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org