You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by lo...@apache.org on 2023/07/13 02:37:12 UTC

[beam] branch master updated: [#27435] Describe the Source and Sink transform and urns in the protos. (#27437)

This is an automated email from the ASF dual-hosted git repository.

lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
     new 0fb46f0e28d [#27435] Describe the Source and Sink transform and urns in the protos. (#27437)
0fb46f0e28d is described below

commit 0fb46f0e28d4ba14a66889fed50d540d13042289
Author: Robert Burke <lo...@users.noreply.github.com>
AuthorDate: Wed Jul 12 19:37:03 2023 -0700

    [#27435] Describe the Source and Sink transform and urns in the protos. (#27437)
    
    * [#27435] Describe source and sink transforms.
    
    * Add underscores.
    
    * re-order because
    
    * Move to fn_api.proto
    
    * rm space
    
    * Describe the enum messages.
    
    * import beam_urn correctly
    
    * robertwb's pr comments & suggestions
    
    ---------
    
    Co-authored-by: lostluck <13...@users.noreply.github.com>
---
 .../beam/model/fn_execution/v1/beam_fn_api.proto   | 46 ++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto b/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto
index efa37ea996e..66d144ab231 100644
--- a/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto
@@ -46,6 +46,52 @@ import "google/protobuf/struct.proto";
 import "google/protobuf/timestamp.proto";
 import "google/protobuf/duration.proto";
 
+
+// Describes transforms necessary to execute Beam over the FnAPI but are
+// implementation details rather than part of the core model.
+message FnApiTransforms {
+  enum Runner {
+    // DataSource is a Root Transform, and a source of data for downstream
+    // transforms in the same ProcessBundleDescriptor. 
+    // It represents a stream of values coming in from an external source/over
+    // a data channel, typically from the runner. It's not the PCollection itself
+    // but a description of how to get the portion of the PCollection for a given
+    // bundle.
+    //
+    // The DataSource transform is implemented in each SDK and not explicitly
+    // provided during pipeline construction. A runner inserts the transform
+    // in ProcessBundleDescriptors to indicate where the bundle
+    // can retrieve data for an associated ProcessBundleRequest.
+    // Data for the same request will be retrieved with the matching instruction ID,
+    // and transform ID determined by the runner.
+    //
+    // The DataSource transform will take a stream of bytes from the remote
+    // source for the matching instruction ID and decode them as windowed
+    // values using the provided coder ID, which must be a windowed value coder.
+    //
+    // Payload: RemoteGrpcPort
+    DATA_SOURCE = 0 [(org.apache.beam.model.pipeline.v1.beam_urn) = "beam:runner:source:v1"];
+
+    // DataSink is a transform that sends PCollection elements to a remote
+    // port using the Data API.
+    //
+    // The DataSink transform is implemented in each SDK and not explicitly
+    // provided during pipeline construction. A runner inserts the transform in
+    // ProcessBundleDescriptors to indicate where the bundle can send
+    // data for each associated ProcessBundleRequest. Data for the same
+    // request will be sent with the matching instruction ID and transform ID.
+    // Each PCollection that exits the ProcessBundleDescriptor subgraph will have
+    // it's own DataSink, keyed by a transform ID determined by the runner.
+    //  
+    // The DataSink will take in a stream of elements for a given instruction ID
+    // and encode them for transmission to the remote sink. The coder ID must be
+    // for a windowed value coder.
+    //
+    // Payload: RemoteGrpcPort
+    DATA_SINK = 1 [(org.apache.beam.model.pipeline.v1.beam_urn) = "beam:runner:sink:v1"];
+  }
+}
+
 // A descriptor for connecting to a remote port using the Beam Fn Data API.
 // Allows for communication between two environments (for example between the
 // runner and the SDK).