You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "lostluck (via GitHub)" <gi...@apache.org> on 2023/07/10 19:39:43 UTC

[GitHub] [beam] lostluck opened a new pull request, #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

lostluck opened a new pull request, #27437:
URL: https://github.com/apache/beam/pull/27437

   Fill in a long standing gap in the proto documentation, by explicitly including the source and sink transforms among the StandardPTransform primitives.
   
   I suspect a reason it went on so long without is that the Urns are largely in the Runner Proto, but everything about the transform is tied to the FnAPI implementation.
   
   Since we no longer expect alternative FnAPI impelementations in Beam V2 that are not GRPC based, it's time to fill that gap.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck merged pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck merged PR #27437:
URL: https://github.com/apache/beam/pull/27437


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1631074451

   Agree. And yes put them in fn api.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1631143751

   Run Python_PVR_Flink PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1632584123

   Kicked the tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1629619691

   What I've put down is better than nothing, so I'm mostly concerned about inaccuracies based on my understanding from the Go SDK and implementing Prism.
   
   The goal isn't to be pedantically precise, but convey the necessary high level view that will enable someone to actually implement the transforms in an SDK, and have it work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on a diff in pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #27437:
URL: https://github.com/apache/beam/pull/27437#discussion_r1259853473


##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto:
##########
@@ -327,6 +327,46 @@ message StandardPTransforms {
     //
     // Payload: none
     TO_STRING = 8 [(beam_urn) = "beam:transform:to_string:v1"];
+
+    // Proto3 requires enum values to be in order, otherwise these would be
+    // at the top of the primitive transforms.
+
+    // DataSource is a Root Transform, and a source of data for downstream
+    // transforms in the same beam_fn_api.ProcessBundleDescriptor. 
+    // It represents a logical PCollection.
+    //
+    // The DataSource transform is implemented in each SDK but not explicitly
+    // provided during pipeline construction. A runner inserts the transform
+    // in beam_fn_api.ProcessBundleDescriptors to indicate where the bundle
+    // can retrieve data for an associated beam_fn_api.ProcessBundleRequest.
+    // Data for the same request will be retrieved with the matching instruction ID,
+    // and transform ID determined by the runner.
+    //
+    // The DataSource transform will take a stream of bytes from the remote
+    // source for the matching instruction ID and decode them as windowed
+    // values using the provided coder ID, which must be for a WINDOWED_VALUE or
+    // PARAM_WINDOWED_VALUE coder.
+    //
+    // Payload: beam_fn_api.RemoteGrpcPort
+    DATA_SOURCE = 9 [(beam_urn) = "beam:runner:source:v1"];

Review Comment:
   I think the reason this isn't in the runner API proto is that it is a fn api detail, not a transform that has any meaning at the whole-graph level.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on a diff in pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #27437:
URL: https://github.com/apache/beam/pull/27437#discussion_r1261730675


##########
model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto:
##########
@@ -46,6 +46,50 @@ import "google/protobuf/struct.proto";
 import "google/protobuf/timestamp.proto";
 import "google/protobuf/duration.proto";
 
+
+// Describes transforms necessary to execute Beam over the FnAPI but are
+// implementation details rather than part of the core model.
+message FnApiTransforms {
+  enum Runner {
+    // DataSource is a Root Transform, and a source of data for downstream
+    // transforms in the same ProcessBundleDescriptor. 
+    // It represents a logical PCollection.
+    //
+    // The DataSource transform is implemented in each SDK but not explicitly
+    // provided during pipeline construction. A runner inserts the transform
+    // in ProcessBundleDescriptors to indicate where the bundle
+    // can retrieve data for an associated ProcessBundleRequest.
+    // Data for the same request will be retrieved with the matching instruction ID,
+    // and transform ID determined by the runner.
+    //
+    // The DataSource transform will take a stream of bytes from the remote
+    // source for the matching instruction ID and decode them as windowed
+    // values using the provided coder ID, which must be for a WINDOWED_VALUE or
+    // PARAM_WINDOWED_VALUE coder.
+    //
+    // Payload: RemoteGrpcPort
+    DATA_SOURCE = 0 [(org.apache.beam.model.pipeline.v1.beam_urn) = "beam:runner:source:v1"];
+
+    // DataSink is a transform that sends data to a remote port
+    // ProcessBundleDescriptor. It represents a logical PCollection.

Review Comment:
   Done.



##########
model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto:
##########
@@ -46,6 +46,50 @@ import "google/protobuf/struct.proto";
 import "google/protobuf/timestamp.proto";
 import "google/protobuf/duration.proto";
 
+
+// Describes transforms necessary to execute Beam over the FnAPI but are
+// implementation details rather than part of the core model.
+message FnApiTransforms {
+  enum Runner {
+    // DataSource is a Root Transform, and a source of data for downstream
+    // transforms in the same ProcessBundleDescriptor. 
+    // It represents a logical PCollection.

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1633263486

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1629615823

   R: @robertwb @byronellis 
   cc: @kennknowles 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1631137734

   > No idea what "runner" has to do with anything but I'm not here to debate URNs and also that ship has sailed. Code matches URN and makes sense.
   
   Technically I think it represented how an SDK communicates with a Runner. But I agree with you, it's not a great name for the urn namespace, and it's too late to change these two. "beam:fnapi:runner:..." would have been a better prefix at least.  Something to fix in a Beam V3 years from now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] robertwb commented on a diff in pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #27437:
URL: https://github.com/apache/beam/pull/27437#discussion_r1260346360


##########
model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto:
##########
@@ -46,6 +46,50 @@ import "google/protobuf/struct.proto";
 import "google/protobuf/timestamp.proto";
 import "google/protobuf/duration.proto";
 
+
+// Describes transforms necessary to execute Beam over the FnAPI but are
+// implementation details rather than part of the core model.
+message FnApiTransforms {
+  enum Runner {
+    // DataSource is a Root Transform, and a source of data for downstream
+    // transforms in the same ProcessBundleDescriptor. 
+    // It represents a logical PCollection.

Review Comment:
   I might say "It represents reading a PCollection from an external source (typically the Runner)" or "represents a stream of values coming in from an external source/over a data channel" as it's not the PCollection itself, but rather a descriptor of how to get the PCollection. 



##########
model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto:
##########
@@ -46,6 +46,50 @@ import "google/protobuf/struct.proto";
 import "google/protobuf/timestamp.proto";
 import "google/protobuf/duration.proto";
 
+
+// Describes transforms necessary to execute Beam over the FnAPI but are
+// implementation details rather than part of the core model.
+message FnApiTransforms {
+  enum Runner {
+    // DataSource is a Root Transform, and a source of data for downstream
+    // transforms in the same ProcessBundleDescriptor. 
+    // It represents a logical PCollection.
+    //
+    // The DataSource transform is implemented in each SDK but not explicitly
+    // provided during pipeline construction. A runner inserts the transform
+    // in ProcessBundleDescriptors to indicate where the bundle
+    // can retrieve data for an associated ProcessBundleRequest.
+    // Data for the same request will be retrieved with the matching instruction ID,
+    // and transform ID determined by the runner.
+    //
+    // The DataSource transform will take a stream of bytes from the remote
+    // source for the matching instruction ID and decode them as windowed
+    // values using the provided coder ID, which must be for a WINDOWED_VALUE or
+    // PARAM_WINDOWED_VALUE coder.
+    //
+    // Payload: RemoteGrpcPort
+    DATA_SOURCE = 0 [(org.apache.beam.model.pipeline.v1.beam_urn) = "beam:runner:source:v1"];
+
+    // DataSink is a transform that sends data to a remote port
+    // ProcessBundleDescriptor. It represents a logical PCollection.

Review Comment:
   "remote port ProcessBundleDescriptor?"
   
   Maybe "DataSink is a transform that sends PCollection elements to a remote port using the Data API."



##########
model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto:
##########
@@ -46,6 +46,50 @@ import "google/protobuf/struct.proto";
 import "google/protobuf/timestamp.proto";
 import "google/protobuf/duration.proto";
 
+
+// Describes transforms necessary to execute Beam over the FnAPI but are
+// implementation details rather than part of the core model.
+message FnApiTransforms {
+  enum Runner {
+    // DataSource is a Root Transform, and a source of data for downstream
+    // transforms in the same ProcessBundleDescriptor. 
+    // It represents a logical PCollection.
+    //
+    // The DataSource transform is implemented in each SDK but not explicitly
+    // provided during pipeline construction. A runner inserts the transform
+    // in ProcessBundleDescriptors to indicate where the bundle
+    // can retrieve data for an associated ProcessBundleRequest.
+    // Data for the same request will be retrieved with the matching instruction ID,
+    // and transform ID determined by the runner.
+    //
+    // The DataSource transform will take a stream of bytes from the remote
+    // source for the matching instruction ID and decode them as windowed
+    // values using the provided coder ID, which must be for a WINDOWED_VALUE or
+    // PARAM_WINDOWED_VALUE coder.

Review Comment:
   We may allow other kinds of acceptable coders in the future, but you could say "must be a windowed value coder" generically.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] codecov[bot] commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1629674477

   ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/27437?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#27437](https://app.codecov.io/gh/apache/beam/pull/27437?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (ec38b59) into [master](https://app.codecov.io/gh/apache/beam/commit/fcde1d15b29b3ef41880d6631dfdd241952a6adf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (fcde1d1) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #27437      +/-   ##
   ==========================================
   - Coverage   71.32%   71.31%   -0.01%     
   ==========================================
     Files         862      862              
     Lines      105205   105205              
   ==========================================
   - Hits        75035    75032       -3     
   - Misses      28621    28624       +3     
     Partials     1549     1549              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `80.51% <ø> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   [see 4 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/27437/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1633281312

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1633334088

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1630973808

   I think this touches on a philosophical difference. In my mind, the pipeline proto / runner API is for authoring parallel processing DAGs, while the Fn API is for invoking a UDF server. They have things in common but transforms should generally _not_ be punned between the two. Of course, transforms _have_ been punned between the two, leading to much confusion. In Dataflow v1 they were not punned, and it was good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1632575387

   ... yea and this is why keep namespaces short and sweet, like `beam:source:v1` or at most namespace it into `beam:transform:source:v1`. Code that looks legacy from day 1 makes me :-(


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1633224338

   Updated based on Robert's comments. Will merge once tests are passing again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1631021185

   I'm happy to put them in beam_fn_api.proto instead. I'm just tired of these being critical to function and being undiscoverable without looking at other SDKs.
   
   It should be possible to construct a compliant Beam SDK without looking at how another SDK did it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1629617627

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #27437:
URL: https://github.com/apache/beam/pull/27437#issuecomment-1631412869

   I have an approval, and the example failure unrelated. I'll merge in  tomorrow morning if there are no objections / additional wordsmithing from other readers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on a diff in pull request #27437: [#27435] Describe the Source and Sink transform and urns in the protos.

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #27437:
URL: https://github.com/apache/beam/pull/27437#discussion_r1261729462


##########
model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto:
##########
@@ -46,6 +46,50 @@ import "google/protobuf/struct.proto";
 import "google/protobuf/timestamp.proto";
 import "google/protobuf/duration.proto";
 
+
+// Describes transforms necessary to execute Beam over the FnAPI but are
+// implementation details rather than part of the core model.
+message FnApiTransforms {
+  enum Runner {
+    // DataSource is a Root Transform, and a source of data for downstream
+    // transforms in the same ProcessBundleDescriptor. 
+    // It represents a logical PCollection.
+    //
+    // The DataSource transform is implemented in each SDK but not explicitly
+    // provided during pipeline construction. A runner inserts the transform
+    // in ProcessBundleDescriptors to indicate where the bundle
+    // can retrieve data for an associated ProcessBundleRequest.
+    // Data for the same request will be retrieved with the matching instruction ID,
+    // and transform ID determined by the runner.
+    //
+    // The DataSource transform will take a stream of bytes from the remote
+    // source for the matching instruction ID and decode them as windowed
+    // values using the provided coder ID, which must be for a WINDOWED_VALUE or
+    // PARAM_WINDOWED_VALUE coder.

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org