You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by lo...@apache.org on 2023/07/13 02:37:12 UTC
[beam] branch master updated: [#27435] Describe the Source and Sink transform and urns in the protos. (#27437)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 0fb46f0e28d [#27435] Describe the Source and Sink transform and urns in the protos. (#27437)
0fb46f0e28d is described below
commit 0fb46f0e28d4ba14a66889fed50d540d13042289
Author: Robert Burke <lo...@users.noreply.github.com>
AuthorDate: Wed Jul 12 19:37:03 2023 -0700
[#27435] Describe the Source and Sink transform and urns in the protos. (#27437)
* [#27435] Describe source and sink transforms.
* Add underscores.
* re-order because
* Move to fn_api.proto
* rm space
* Describe the enum messages.
* import beam_urn correctly
* robertwb's pr comments & suggestions
---------
Co-authored-by: lostluck <13...@users.noreply.github.com>
---
.../beam/model/fn_execution/v1/beam_fn_api.proto | 46 ++++++++++++++++++++++
1 file changed, 46 insertions(+)
diff --git a/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto b/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto
index efa37ea996e..66d144ab231 100644
--- a/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto
@@ -46,6 +46,52 @@ import "google/protobuf/struct.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";
+
+// Describes transforms necessary to execute Beam over the FnAPI but are
+// implementation details rather than part of the core model.
+message FnApiTransforms {
+ enum Runner {
+ // DataSource is a Root Transform, and a source of data for downstream
+ // transforms in the same ProcessBundleDescriptor.
+ // It represents a stream of values coming in from an external source/over
+ // a data channel, typically from the runner. It's not the PCollection itself
+ // but a description of how to get the portion of the PCollection for a given
+ // bundle.
+ //
+ // The DataSource transform is implemented in each SDK and not explicitly
+ // provided during pipeline construction. A runner inserts the transform
+ // in ProcessBundleDescriptors to indicate where the bundle
+ // can retrieve data for an associated ProcessBundleRequest.
+ // Data for the same request will be retrieved with the matching instruction ID,
+ // and transform ID determined by the runner.
+ //
+ // The DataSource transform will take a stream of bytes from the remote
+ // source for the matching instruction ID and decode them as windowed
+ // values using the provided coder ID, which must be a windowed value coder.
+ //
+ // Payload: RemoteGrpcPort
+ DATA_SOURCE = 0 [(org.apache.beam.model.pipeline.v1.beam_urn) = "beam:runner:source:v1"];
+
+ // DataSink is a transform that sends PCollection elements to a remote
+ // port using the Data API.
+ //
+ // The DataSink transform is implemented in each SDK and not explicitly
+ // provided during pipeline construction. A runner inserts the transform in
+ // ProcessBundleDescriptors to indicate where the bundle can send
+ // data for each associated ProcessBundleRequest. Data for the same
+ // request will be sent with the matching instruction ID and transform ID.
+ // Each PCollection that exits the ProcessBundleDescriptor subgraph will have
+ // it's own DataSink, keyed by a transform ID determined by the runner.
+ //
+ // The DataSink will take in a stream of elements for a given instruction ID
+ // and encode them for transmission to the remote sink. The coder ID must be
+ // for a windowed value coder.
+ //
+ // Payload: RemoteGrpcPort
+ DATA_SINK = 1 [(org.apache.beam.model.pipeline.v1.beam_urn) = "beam:runner:sink:v1"];
+ }
+}
+
// A descriptor for connecting to a remote port using the Beam Fn Data API.
// Allows for communication between two environments (for example between the
// runner and the SDK).