[GitHub] [beam] damccorm opened a new issue, #17852: Data-dependent sinks

[GitBox <gi...@apache.org>]

damccorm opened a new issue, #17852:
URL: https://github.com/apache/beam/issues/17852

   Current sink API writes all data to a single destination, but there are many use cases where different pieces of data need to be routed to different destinations where the set of destinations is data-dependent (so can't be implemented with a Partition transform).
   
   One internally discussed proposal was an API of the form:
   ```
   
   PCollection<Void> PCollection<T>.apply(
       Write.using(DoFn<T, SinkT> where,
                   MapFn<SinkT,
   WriteOperation<WriteResultT, T>> how)
   
   ```
   
   
   so an item T gets written to a destination (or multiple destinations) determined by "where"; and the writing strategy is determined by "how" that produces a WriteOperation (current API - global init/write/global finalize hooks) for any given destination.
   
   This API also has other benefits:
   * allows the SinkT to be computed dynamically (in "where"), rather than specified at pipeline construction time
   * removes the necessity for a Sink class entirely
   * is sequenceable w.r.t. downstream transforms (you can stick transforms onto the returned PCollection<Void\>, while the current Write.to() returns a PDone)
   
   Imported from Jira [BEAM-92](https://issues.apache.org/jira/browse/BEAM-92). Original Jira may contain additional context.
   Reported by: jkff.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm closed issue #17852: Data-dependent sinks

[GitBox <gi...@apache.org>]

damccorm closed issue #17852: Data-dependent sinks
URL: https://github.com/apache/beam/issues/17852


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org