You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "prodriguezdefino (via GitHub)" <gi...@apache.org> on 2023/11/29 23:19:36 UTC

[PR] Generalize file based source output capabilities [beam]

prodriguezdefino opened a new pull request, #29566:
URL: https://github.com/apache/beam/pull/29566

   Currently, when using a file based source implementation to read data from files we have 2 options: read only the content of the each file into a PCollection or read the content of a file, key it with the file name and put it into a PCollection. While this cover a large bulk of the desired read operations, we can generalize the current implementation to allow a much larger space of possibilities for the users.
   
   By introducing a serializable lambda as a parameter for the `ReadAllViaFileBasedSource` class, and a data carrier adapter to simplify the makeOutput method arguments with an arbitrarily provided lambda, we can enable users to make interesting decisions about how to treat the data read from particular files by making more information accessible about the files their data was extracted from. 
   
   This change adds such generalization to the TextIO PTransform and an example that extracts not only a file name with the data line contained in the file, but also the originally file size which can be used to predicate, for example, if the destination file should be partitioned in many given the input one size. 
   
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967336923

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1913138465

   Reminder, please take a look at this pr: @Abacn @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1918989406

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @damondouglas for label java.
   R: @damondouglas for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1855739970

   Reminder, please take a look at this pr: @bvolpato @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1878573575

   Reminder, please take a look at this pr: @damondouglas @damondouglas 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1893625576

   Reminder, please take a look at this pr: @robertwb @johnjcasey 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #29566: Generalize file based source output capabilities
URL: https://github.com/apache/beam/pull/29566


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-2184973869

   This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1941382305

   Reminder, please take a look at this pr: @damondouglas @damondouglas 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1882963790

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @robertwb for label java.
   R: @johnjcasey for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1841541894

   Run Java_IOs_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967277975

   Good day, @prodriguezdefino, Is this PR still active? Would you be able to investigate the two failing checks? Please let me know if you need anything and thank you again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on code in PR #29566:
URL: https://github.com/apache/beam/pull/29566#discussion_r1475338512


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
  * <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
  * FileIO#readMatches()}.
  */
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
-  public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+  private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+  protected ReadAllViaFileBasedSource(
       long desiredBundleSizeBytes,
       SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-      Coder<T> coder) {
+      Coder<K> coder,
+      SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
     super(
         desiredBundleSizeBytes,
         createSource,
         coder,
         DEFAULT_USES_RESHUFFLE,
-        new ReadFileRangesFnExceptionHandler());
+        new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+    this.outputFn = outputFn;
   }
 
-  public ReadAllViaFileBasedSource(
+  protected ReadAllViaFileBasedSource(

Review Comment:
   Would the reduction in visibility break others code who may already depend on this?



##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
  * <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
  * FileIO#readMatches()}.
  */
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
-  public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+  private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+  protected ReadAllViaFileBasedSource(
       long desiredBundleSizeBytes,
       SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-      Coder<T> coder) {
+      Coder<K> coder,
+      SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
     super(
         desiredBundleSizeBytes,
         createSource,
         coder,
         DEFAULT_USES_RESHUFFLE,
-        new ReadFileRangesFnExceptionHandler());
+        new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+    this.outputFn = outputFn;
   }
 
-  public ReadAllViaFileBasedSource(
+  protected ReadAllViaFileBasedSource(
       long desiredBundleSizeBytes,
       SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-      Coder<T> coder,
+      Coder<K> coder,
       boolean usesReshuffle,
-      ReadFileRangesFnExceptionHandler exceptionHandler) {
+      ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler,
+      SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
     super(desiredBundleSizeBytes, createSource, coder, usesReshuffle, exceptionHandler);
+    this.outputFn = outputFn;
+  }
+
+  public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+      long desiredBundleSizeBytes,
+      SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+      Coder<InputT> coder,
+      boolean usesReshuffle,
+      ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler) {
+    return new ReadAllViaFileBasedSource<>(
+        desiredBundleSizeBytes,
+        createSource,
+        coder,
+        usesReshuffle,
+        exceptionHandler,
+        outputArguments -> outputArguments.reader().getCurrent());
+  }
+
+  public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+      long desiredBundleSizeBytes,
+      SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+      Coder<InputT> coder) {
+    return create(
+        desiredBundleSizeBytes,
+        createSource,
+        coder,
+        outputArguments -> outputArguments.reader().getCurrent());
+  }
+
+  public static <InputT, OutputT> ReadAllViaFileBasedSource<InputT, OutputT> create(
+      long desiredBundleSizeBytes,
+      SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+      Coder<OutputT> coder,
+      SerializableFunction<OutputContextFromFile<InputT>, OutputT> outputFn) {
+    return new ReadAllViaFileBasedSource<>(desiredBundleSizeBytes, createSource, coder, outputFn);
   }
 
   @Override
-  protected DoFn<KV<ReadableFile, OffsetRange>, T> readRangesFn() {
-    return new ReadFileRangesFn<>(createSource, exceptionHandler);
+  protected DoFn<KV<ReadableFile, OffsetRange>, K> readRangesFn() {
+    return new ReadFileRangesFn<>(outputFn, createSource, exceptionHandler);
   }
 
-  private static class ReadFileRangesFn<T> extends AbstractReadFileRangesFn<T, T> {
+  private static class ReadFileRangesFn<T, K> extends AbstractReadFileRangesFn<T, K> {
+    private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
     public ReadFileRangesFn(
+        final SerializableFunction<OutputContextFromFile<T>, K> outputFn,
         final SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-        final ReadFileRangesFnExceptionHandler exceptionHandler) {
+        final ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler

Review Comment:
   Would changing this signature of a public method break others code that already depends on this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1833268429

   assign set of reviewers


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1833171066

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967331829

   Sorry on the delay, checking on this now. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-2169498263

   This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1833270082

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @bvolpato for label java.
   R: @ahmedabu98 for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1833217233

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1842085568

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on code in PR #29566:
URL: https://github.com/apache/beam/pull/29566#discussion_r1476626897


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
  * <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
  * FileIO#readMatches()}.
  */
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
-  public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+  private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+  protected ReadAllViaFileBasedSource(
       long desiredBundleSizeBytes,
       SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-      Coder<T> coder) {
+      Coder<K> coder,
+      SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
     super(
         desiredBundleSizeBytes,
         createSource,
         coder,
         DEFAULT_USES_RESHUFFLE,
-        new ReadFileRangesFnExceptionHandler());
+        new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+    this.outputFn = outputFn;
   }
 
-  public ReadAllViaFileBasedSource(
+  protected ReadAllViaFileBasedSource(
       long desiredBundleSizeBytes,
       SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-      Coder<T> coder,
+      Coder<K> coder,
       boolean usesReshuffle,
-      ReadFileRangesFnExceptionHandler exceptionHandler) {
+      ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler,
+      SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
     super(desiredBundleSizeBytes, createSource, coder, usesReshuffle, exceptionHandler);
+    this.outputFn = outputFn;
+  }
+
+  public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+      long desiredBundleSizeBytes,
+      SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+      Coder<InputT> coder,
+      boolean usesReshuffle,
+      ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler) {
+    return new ReadAllViaFileBasedSource<>(
+        desiredBundleSizeBytes,
+        createSource,
+        coder,
+        usesReshuffle,
+        exceptionHandler,
+        outputArguments -> outputArguments.reader().getCurrent());
+  }
+
+  public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+      long desiredBundleSizeBytes,
+      SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+      Coder<InputT> coder) {
+    return create(
+        desiredBundleSizeBytes,
+        createSource,
+        coder,
+        outputArguments -> outputArguments.reader().getCurrent());
+  }
+
+  public static <InputT, OutputT> ReadAllViaFileBasedSource<InputT, OutputT> create(
+      long desiredBundleSizeBytes,
+      SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+      Coder<OutputT> coder,
+      SerializableFunction<OutputContextFromFile<InputT>, OutputT> outputFn) {
+    return new ReadAllViaFileBasedSource<>(desiredBundleSizeBytes, createSource, coder, outputFn);
   }
 
   @Override
-  protected DoFn<KV<ReadableFile, OffsetRange>, T> readRangesFn() {
-    return new ReadFileRangesFn<>(createSource, exceptionHandler);
+  protected DoFn<KV<ReadableFile, OffsetRange>, K> readRangesFn() {
+    return new ReadFileRangesFn<>(outputFn, createSource, exceptionHandler);
   }
 
-  private static class ReadFileRangesFn<T> extends AbstractReadFileRangesFn<T, T> {
+  private static class ReadFileRangesFn<T, K> extends AbstractReadFileRangesFn<T, K> {
+    private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
     public ReadFileRangesFn(
+        final SerializableFunction<OutputContextFromFile<T>, K> outputFn,
         final SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-        final ReadFileRangesFnExceptionHandler exceptionHandler) {
+        final ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler

Review Comment:
   This change obey the movement of that static class (`ReadFileRangesFnExceptionHandler`) to the parent class in the hierarchy. In master the code has a weird situation where a parent class depends on a static class implemented in one of the child classes. 
   
   I moved `ReadFileRangesFnExceptionHandler` to the top of the hierarchy to avoid the double reference, and changed the references in the Beam framework. Again this may break user code if they make direct reference to this static class, so I'm happy to revert that change if preferred. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1964590702

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "RCQE-bot (via GitHub)" <gi...@apache.org>.
RCQE-bot commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-2058847972

   ## RCQE-bot: Commit Quality Assessment  πŸ”Ž 
   I've carefully evaluated the recent commits, and here's the current state of the commit quality: 
   ### Metrics πŸ“Š  
   - Previous Commits: diff hunk in file [sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java](https://github.com/apache/beam/pull/29566/files/349afba29fde04b3761f0d2956a84ed465355ef2#diff-8cecdc624834472e9bbcc9c2f5a129224d36d9f84a8e3316fe774d232af400e8) in commit [7e1eb23b904059c23dcd3d6fb13c39ce13a27930](https://github.com/apache/beam/pull/29566/commits/7e1eb23b904059c23dcd3d6fb13c39ce13a27930)
   - Code Review: Would changing this signature of a public method break others code that already depends on this?
   ### Suggestions πŸ’‘ 
   - Need Review Again 🚫 : The commits do not currently meet the quality standards. I recommend not merging them into the main branch until the above suggestions have been implemented and a subsequent review confirms the quality. Please address the points mentioned and submit the commits for another round of review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1860329723

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @kennknowles for label java.
   R: @johnjcasey for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1869499056

   Reminder, please take a look at this pr: @kennknowles @johnjcasey 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1843277476

   This is a really interesting enhancement, but it goes to the core of a Beam transform. Do you have a design document for this? I want to make sure the community sees the trade offs here before we go too far into implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1964587970

   This sounds like a good change. 
   
   Formally adding R: @damondouglas for follow-up. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on code in PR #29566:
URL: https://github.com/apache/beam/pull/29566#discussion_r1504737909


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
  * <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
  * FileIO#readMatches()}.
  */
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
-  public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+  private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+  protected ReadAllViaFileBasedSource(
       long desiredBundleSizeBytes,
       SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-      Coder<T> coder) {
+      Coder<K> coder,
+      SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
     super(
         desiredBundleSizeBytes,
         createSource,
         coder,
         DEFAULT_USES_RESHUFFLE,
-        new ReadFileRangesFnExceptionHandler());
+        new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+    this.outputFn = outputFn;
   }
 
-  public ReadAllViaFileBasedSource(
+  protected ReadAllViaFileBasedSource(
       long desiredBundleSizeBytes,
       SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-      Coder<T> coder,
+      Coder<K> coder,
       boolean usesReshuffle,
-      ReadFileRangesFnExceptionHandler exceptionHandler) {
+      ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler,
+      SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
     super(desiredBundleSizeBytes, createSource, coder, usesReshuffle, exceptionHandler);
+    this.outputFn = outputFn;
+  }
+
+  public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+      long desiredBundleSizeBytes,
+      SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+      Coder<InputT> coder,
+      boolean usesReshuffle,
+      ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler exceptionHandler) {
+    return new ReadAllViaFileBasedSource<>(
+        desiredBundleSizeBytes,
+        createSource,
+        coder,
+        usesReshuffle,
+        exceptionHandler,
+        outputArguments -> outputArguments.reader().getCurrent());
+  }
+
+  public static <InputT> ReadAllViaFileBasedSource<InputT, InputT> create(
+      long desiredBundleSizeBytes,
+      SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+      Coder<InputT> coder) {
+    return create(
+        desiredBundleSizeBytes,
+        createSource,
+        coder,
+        outputArguments -> outputArguments.reader().getCurrent());
+  }
+
+  public static <InputT, OutputT> ReadAllViaFileBasedSource<InputT, OutputT> create(
+      long desiredBundleSizeBytes,
+      SerializableFunction<String, ? extends FileBasedSource<InputT>> createSource,
+      Coder<OutputT> coder,
+      SerializableFunction<OutputContextFromFile<InputT>, OutputT> outputFn) {
+    return new ReadAllViaFileBasedSource<>(desiredBundleSizeBytes, createSource, coder, outputFn);
   }
 
   @Override
-  protected DoFn<KV<ReadableFile, OffsetRange>, T> readRangesFn() {
-    return new ReadFileRangesFn<>(createSource, exceptionHandler);
+  protected DoFn<KV<ReadableFile, OffsetRange>, K> readRangesFn() {
+    return new ReadFileRangesFn<>(outputFn, createSource, exceptionHandler);
   }
 
-  private static class ReadFileRangesFn<T> extends AbstractReadFileRangesFn<T, T> {
+  private static class ReadFileRangesFn<T, K> extends AbstractReadFileRangesFn<T, K> {
+    private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
     public ReadFileRangesFn(
+        final SerializableFunction<OutputContextFromFile<T>, K> outputFn,
         final SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-        final ReadFileRangesFnExceptionHandler exceptionHandler) {
+        final ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler

Review Comment:
   @damondouglas let me know if you want me to revert the movement of `ReadFileRangesFnExceptionHandler` to the top of the hierarchy. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967337231

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1967557849

   Run Java_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1962345087

   Reminder, please take a look at this pr: @robertwb @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1871110685

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @damondouglas for label java.
   R: @damondouglas for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1841972001

   fixes #29627


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1841697835

   pull_request


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on code in PR #29566:
URL: https://github.com/apache/beam/pull/29566#discussion_r1476620928


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java:
##########
@@ -35,59 +35,114 @@
  * <p>To obtain the collection of {@link ReadableFile} from a filepattern, use {@link
  * FileIO#readMatches()}.
  */
-public class ReadAllViaFileBasedSource<T> extends ReadAllViaFileBasedSourceTransform<T, T> {
-  public ReadAllViaFileBasedSource(
+public class ReadAllViaFileBasedSource<T, K> extends ReadAllViaFileBasedSourceTransform<T, K> {
+
+  private final SerializableFunction<OutputContextFromFile<T>, K> outputFn;
+
+  protected ReadAllViaFileBasedSource(
       long desiredBundleSizeBytes,
       SerializableFunction<String, ? extends FileBasedSource<T>> createSource,
-      Coder<T> coder) {
+      Coder<K> coder,
+      SerializableFunction<OutputContextFromFile<T>, K> outputFn) {
     super(
         desiredBundleSizeBytes,
         createSource,
         coder,
         DEFAULT_USES_RESHUFFLE,
-        new ReadFileRangesFnExceptionHandler());
+        new ReadAllViaFileBasedSourceTransform.ReadFileRangesFnExceptionHandler());
+    this.outputFn = outputFn;
   }
 
-  public ReadAllViaFileBasedSource(
+  protected ReadAllViaFileBasedSource(

Review Comment:
   I'm happy to revert the visibility change on the constructor signature. 
   The other transforms that uses this class in Beam only accessed the functionality through inheritance.  I understand the potential breakage of other user based transforms that may be impacted by this code change. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1948280091

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @robertwb for label java.
   R: @ahmedabu98 for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1919753414

   Good day, @prodriguezdefino and thank you for contributing! I will be taking a look at this and placed it in my review queue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Generalize file based source output capabilities [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29566:
URL: https://github.com/apache/beam/pull/29566#issuecomment-1900308337

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @Abacn for label java.
   R: @ahmedabu98 for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org