[GitHub] [beam] chamikaramj commented on a change in pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

[GitBox <gi...@apache.org>]

chamikaramj commented on a change in pull request #12760:
URL: https://github.com/apache/beam/pull/12760#discussion_r487354247



##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org