You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/03 16:06:40 UTC
[GitHub] [beam] kennknowles opened a new issue, #17980: Data-dependent sinks
kennknowles opened a new issue, #17980:
URL: https://github.com/apache/beam/issues/17980
Current sink API writes all data to a single destination, but there are many use cases where different pieces of data need to be routed to different destinations where the set of destinations is data-dependent (so can't be implemented with a Partition transform).
One internally discussed proposal was an API of the form:
```
PCollection<Void> PCollection<T>.apply(
Write.using(DoFn<T, SinkT> where,
MapFn<SinkT,
WriteOperation<WriteResultT, T>> how)
```
so an item T gets written to a destination (or multiple destinations) determined by "where"; and the writing strategy is determined by "how" that produces a WriteOperation (current API - global init/write/global finalize hooks) for any given destination.
This API also has other benefits:
* allows the SinkT to be computed dynamically (in "where"), rather than specified at pipeline construction time
* removes the necessity for a Sink class entirely
* is sequenceable w.r.t. downstream transforms (you can stick transforms onto the returned PCollection<Void\>, while the current Write.to() returns a PDone)
Imported from Jira [BEAM-92](https://issues.apache.org/jira/browse/BEAM-92). Original Jira may contain additional context.
Reported by: jkff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org