You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/02 16:48:18 UTC

[GitHub] [beam] chamikaramj opened a new pull request #12760: Adds URNs and payloads to PubSub transforms to allow runner-native overrides

chamikaramj opened a new pull request #12760:
URL: https://github.com/apache/beam/pull/12760


   WIP - not ready for review
   
   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](htt
 ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_
 Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_P
 ostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   ![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg)
   ![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg)
   ![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `0.00%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #12760   +/-   ##
   =======================================
     Coverage   40.38%   40.39%           
   =======================================
     Files         451      451           
     Lines       53721    53768   +47     
   =======================================
   + Hits        21696    21720   +24     
   - Misses      32025    32048   +23     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `27.92% <0.00%> (-1.14%)` | :arrow_down: |
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `24.34% <0.00%> (-0.66%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...72058e4](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `42.02%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12760       +/-   ##
   ===========================================
   + Coverage   40.38%   82.40%   +42.02%     
   ===========================================
     Files         451      451               
     Lines       53721    53898      +177     
   ===========================================
   + Hits        21696    44416    +22720     
   + Misses      32025     9482    -22543     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+51.57%)` | :arrow_up: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | ... and [271 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...72058e4](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **decrease** coverage by `0.23%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   - Coverage   40.23%   40.00%   -0.24%     
   ==========================================
     Files         455      457       +2     
     Lines       53726    55560    +1834     
   ==========================================
   + Hits        21619    22224     +605     
   - Misses      32107    33336    +1229     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
   | [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `23.06% <0.00%> (-1.17%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `28.50% <0.00%> (-1.10%)` | :arrow_down: |
   | [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
   | ... and [12 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [2f8ea18...9dd622c](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `0.00%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #12760   +/-   ##
   =======================================
     Coverage   40.38%   40.39%           
   =======================================
     Files         451      451           
     Lines       53721    53768   +47     
   =======================================
   + Hits        21696    21720   +24     
   - Misses      32025    32048   +23     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `27.92% <0.00%> (-1.14%)` | :arrow_down: |
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `24.34% <0.00%> (-0.66%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...c131ca0](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `42.02%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12760       +/-   ##
   ===========================================
   + Coverage   40.38%   82.40%   +42.02%     
   ===========================================
     Files         451      451               
     Lines       53721    53898      +177     
   ===========================================
   + Hits        21696    44416    +22720     
   + Misses      32025     9482    -22543     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+51.57%)` | :arrow_up: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | ... and [271 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...72058e4](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-691513793


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   + Coverage   40.23%   40.28%   +0.04%     
   ==========================================
     Files         455      455              
     Lines       53726    53852     +126     
   ==========================================
   + Hits        21619    21696      +77     
   - Misses      32107    32156      +49     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
   | [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.92% <0.00%> (+0.24%)` | :arrow_up: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `43.73% <0.00%> (+0.76%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `39.22% <0.00%> (+0.94%)` | :arrow_up: |
   | [...s/python/apache\_beam/testing/synthetic\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9zeW50aGV0aWNfcGlwZWxpbmUucHk=) | `23.45% <0.00%> (+2.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [2f8ea18...9dd622c](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **decrease** coverage by `0.23%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   - Coverage   40.23%   40.00%   -0.24%     
   ==========================================
     Files         455      457       +2     
     Lines       53726    55560    +1834     
   ==========================================
   + Hits        21619    22224     +605     
   - Misses      32107    33336    +1229     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
   | [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `23.06% <0.00%> (-1.17%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `28.50% <0.00%> (-1.10%)` | :arrow_down: |
   | [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
   | ... and [12 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [2f8ea18...9dd622c](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `42.02%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12760       +/-   ##
   ===========================================
   + Coverage   40.38%   82.40%   +42.02%     
   ===========================================
     Files         451      451               
     Lines       53721    53898      +177     
   ===========================================
   + Hits        21696    44416    +22720     
   + Misses      32025     9482    -22543     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+51.57%)` | :arrow_up: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | ... and [271 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...72058e4](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-690413846


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `42.02%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12760       +/-   ##
   ===========================================
   + Coverage   40.38%   82.40%   +42.02%     
   ===========================================
     Files         451      451               
     Lines       53721    53898      +177     
   ===========================================
   + Hits        21696    44416    +22720     
   + Misses      32025     9482    -22543     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+51.57%)` | :arrow_up: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | ... and [271 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...72058e4](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-691198263






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on a change in pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on a change in pull request #12760:
URL: https://github.com/apache/beam/pull/12760#discussion_r487354247



##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   + Coverage   40.23%   40.28%   +0.04%     
   ==========================================
     Files         455      455              
     Lines       53726    53852     +126     
   ==========================================
   + Hits        21619    21696      +77     
   - Misses      32107    32156      +49     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
   | [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.92% <0.00%> (+0.24%)` | :arrow_up: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `43.73% <0.00%> (+0.76%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `39.22% <0.00%> (+0.94%)` | :arrow_up: |
   | [...s/python/apache\_beam/testing/synthetic\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9zeW50aGV0aWNfcGlwZWxpbmUucHk=) | `23.45% <0.00%> (+2.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [2f8ea18...9dd622c](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj merged pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj merged pull request #12760:
URL: https://github.com/apache/beam/pull/12760


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `42.02%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12760       +/-   ##
   ===========================================
   + Coverage   40.38%   82.40%   +42.02%     
   ===========================================
     Files         451      451               
     Lines       53721    53898      +177     
   ===========================================
   + Hits        21696    44416    +22720     
   + Misses      32025     9482    -22543     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+51.57%)` | :arrow_up: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | ... and [271 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...72058e4](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-691532731


   Java failure is unrelated - https://issues.apache.org/jira/browse/BEAM-10886


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on a change in pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on a change in pull request #12760:
URL: https://github.com/apache/beam/pull/12760#discussion_r487354247



##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `0.00%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #12760   +/-   ##
   =======================================
     Coverage   40.38%   40.39%           
   =======================================
     Files         451      451           
     Lines       53721    53768   +47     
   =======================================
   + Hits        21696    21720   +24     
   - Misses      32025    32048   +23     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `27.92% <0.00%> (-1.14%)` | :arrow_down: |
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `24.34% <0.00%> (-0.66%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...c131ca0](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **decrease** coverage by `0.23%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   - Coverage   40.23%   40.00%   -0.24%     
   ==========================================
     Files         455      457       +2     
     Lines       53726    55560    +1834     
   ==========================================
   + Hits        21619    22224     +605     
   - Misses      32107    33336    +1229     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
   | [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `23.06% <0.00%> (-1.17%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `28.50% <0.00%> (-1.10%)` | :arrow_down: |
   | [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
   | ... and [12 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [2f8ea18...9dd622c](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/8bae9b3cbe184a153752fd65c3df6341c6944052?el=desc) will **decrease** coverage by `0.38%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   - Coverage   40.38%   40.00%   -0.39%     
   ==========================================
     Files         451      457       +6     
     Lines       53721    55560    +1839     
   ==========================================
   + Hits        21696    22224     +528     
   - Misses      32025    33336    +1311     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `23.06% <0.00%> (-1.22%)` | :arrow_down: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `28.50% <0.00%> (-0.55%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...he\_beam/testing/benchmarks/nexmark/nexmark\_util.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya191dGlsLnB5) | `0.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...eam/testing/benchmarks/nexmark/nexmark\_launcher.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvbmV4bWFya19sYXVuY2hlci5weQ==) | `0.00% <0.00%> (ø)` | |
   | [.../python/apache\_beam/tools/map\_fn\_microbenchmark.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdG9vbHMvbWFwX2ZuX21pY3JvYmVuY2htYXJrLnB5) | `0.00% <0.00%> (ø)` | |
   | [.../python/apache\_beam/tools/coders\_microbenchmark.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdG9vbHMvY29kZXJzX21pY3JvYmVuY2htYXJrLnB5) | `42.85% <0.00%> (ø)` | |
   | [...\_beam/tools/distribution\_counter\_microbenchmark.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdG9vbHMvZGlzdHJpYnV0aW9uX2NvdW50ZXJfbWljcm9iZW5jaG1hcmsucHk=) | `0.00% <0.00%> (ø)` | |
   | ... and [9 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...c131ca0](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   + Coverage   40.23%   40.28%   +0.04%     
   ==========================================
     Files         455      455              
     Lines       53726    53852     +126     
   ==========================================
   + Hits        21619    21696      +77     
   - Misses      32107    32156      +49     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
   | [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.92% <0.00%> (+0.24%)` | :arrow_up: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `43.73% <0.00%> (+0.76%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `39.22% <0.00%> (+0.94%)` | :arrow_up: |
   | [...s/python/apache\_beam/testing/synthetic\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9zeW50aGV0aWNfcGlwZWxpbmUucHk=) | `23.45% <0.00%> (+2.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [2f8ea18...9dd622c](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `42.02%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12760       +/-   ##
   ===========================================
   + Coverage   40.38%   82.40%   +42.02%     
   ===========================================
     Files         451      451               
     Lines       53721    53898      +177     
   ===========================================
   + Hits        21696    44416    +22720     
   + Misses      32025     9482    -22543     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+51.57%)` | :arrow_up: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | ... and [271 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...72058e4](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `42.02%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12760       +/-   ##
   ===========================================
   + Coverage   40.38%   82.40%   +42.02%     
   ===========================================
     Files         451      451               
     Lines       53721    53898      +177     
   ===========================================
   + Hits        21696    44416    +22720     
   + Misses      32025     9482    -22543     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+51.57%)` | :arrow_up: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | ... and [271 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...72058e4](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-691198263






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on a change in pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on a change in pull request #12760:
URL: https://github.com/apache/beam/pull/12760#discussion_r487332677



##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-691198263






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   + Coverage   40.23%   40.28%   +0.04%     
   ==========================================
     Files         455      455              
     Lines       53726    53852     +126     
   ==========================================
   + Hits        21619    21696      +77     
   - Misses      32107    32156      +49     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
   | [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.92% <0.00%> (+0.24%)` | :arrow_up: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `43.73% <0.00%> (+0.76%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `39.22% <0.00%> (+0.94%)` | :arrow_up: |
   | [...s/python/apache\_beam/testing/synthetic\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9zeW50aGV0aWNfcGlwZWxpbmUucHk=) | `23.45% <0.00%> (+2.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [2f8ea18...9dd622c](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689864084


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   + Coverage   40.23%   40.28%   +0.04%     
   ==========================================
     Files         455      455              
     Lines       53726    53852     +126     
   ==========================================
   + Hits        21619    21696      +77     
   - Misses      32107    32156      +49     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
   | [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.92% <0.00%> (+0.24%)` | :arrow_up: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `43.73% <0.00%> (+0.76%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `39.22% <0.00%> (+0.94%)` | :arrow_up: |
   | [...s/python/apache\_beam/testing/synthetic\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9zeW50aGV0aWNfcGlwZWxpbmUucHk=) | `23.45% <0.00%> (+2.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [2f8ea18...9dd622c](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-690548224


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on a change in pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on a change in pull request #12760:
URL: https://github.com/apache/beam/pull/12760#discussion_r487332677



##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `0.00%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #12760   +/-   ##
   =======================================
     Coverage   40.38%   40.39%           
   =======================================
     Files         451      451           
     Lines       53721    53768   +47     
   =======================================
   + Hits        21696    21720   +24     
   - Misses      32025    32048   +23     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `27.92% <0.00%> (-1.14%)` | :arrow_down: |
   | [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `24.34% <0.00%> (-0.66%)` | :arrow_down: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...c131ca0](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] boyuanzz commented on a change in pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

boyuanzz commented on a change in pull request #12760:
URL: https://github.com/apache/beam/pull/12760#discussion_r487332677



##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.

##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       Just curious why we use `string` not `bytes` for serialized fn.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-690413073


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689037944


   R: @robertwb 
   
   CC: @lukecwik 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-691507386


   Retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on a change in pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on a change in pull request #12760:
URL: https://github.com/apache/beam/pull/12760#discussion_r487354247



##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -650,6 +656,54 @@ message WriteFilesPayload {
   map<string, SideInput> side_inputs = 5;
 }
 
+// Payload used by Google Cloud Pub/Sub read transform.
+// This can be used by runners that wish to override Beam Pub/Sub read transform
+// with a native implementation.
+message PubSubReadPayload {
+
+  // Topic to read from. Exactly one of topic or subscription should be set.
+  string topic = 1;
+
+  // Subscription to read from. Exactly one of topic or subscription should be set.
+  string subscription = 2;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 3;
+
+  // Attribute to be used for uniquely identifying messages.
+  string id_attribute = 4;
+
+  // If true, reads Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 5;
+
+  // Parse function for attibutes.
+  string serialized_attribute_fn = 6;
+}
+
+// Payload used by Google Cloud Pub/Sub write transform.
+// This can be used by runners that wish to override Beam Pub/Sub write transform
+// with a native implementation.
+message PubSubWritePayload {
+
+  // Topic to write to.
+  string topic = 1;
+
+  // Attribute that provides element timestamps.
+  string timestamp_attribute = 2;
+
+  // Attribute that uniquely identify messages.
+  string id_attribute = 3;
+
+  // If true, writes Pub/Sub payload as well as attributes. If false, reads only the payload.
+  bool with_attributes = 4;
+
+  // Parse function for attributes.
+  string serialized_attribute_fn = 5;

Review comment:
       This is to be provided as JSON serialized bytes. Slightly updated the documentation.
   This is to be compatible with Dataflow implementation which is the only runner that will perform a native override of PubSub transforms.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-691507512


   Retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689864226


   Retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj merged pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj merged pull request #12760:
URL: https://github.com/apache/beam/pull/12760






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/57055262e7a6bff447eef2df1e6efcda754939ca?el=desc) will **increase** coverage by `42.02%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #12760       +/-   ##
   ===========================================
   + Coverage   40.38%   82.40%   +42.02%     
   ===========================================
     Files         451      451               
     Lines       53721    53898      +177     
   ===========================================
   + Hits        21696    44416    +22720     
   + Misses      32025     9482    -22543     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+51.57%)` | :arrow_up: |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [sdks/python/apache\_beam/coders/slow\_stream.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Nsb3dfc3RyZWFtLnB5) | `92.43% <0.00%> (+1.68%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/profiler.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvZmlsZXIucHk=) | `32.11% <0.00%> (+1.83%)` | :arrow_up: |
   | [...on/apache\_beam/runners/direct/sdf\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvc2RmX2RpcmVjdF9ydW5uZXIucHk=) | `36.21% <0.00%> (+2.46%)` | :arrow_up: |
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.25% <0.00%> (+2.64%)` | :arrow_up: |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.95% <0.00%> (+3.19%)` | :arrow_up: |
   | [sdks/python/apache\_beam/utils/proto\_utils.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcHJvdG9fdXRpbHMucHk=) | `63.07% <0.00%> (+4.61%)` | :arrow_up: |
   | [...pache\_beam/examples/cookbook/bigquery\_tornadoes.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svYmlncXVlcnlfdG9ybmFkb2VzLnB5) | `45.00% <0.00%> (+5.00%)` | :arrow_up: |
   | ... and [271 more](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [5705526...72058e4](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj merged pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj merged pull request #12760:
URL: https://github.com/apache/beam/pull/12760






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] edited a comment on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

codecov[bot] edited a comment on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-689053328


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=h1) Report
   > Merging [#12760](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/2f8ea18341ba602b6c0c776bf253f605acfdd4dd?el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `58.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12760/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #12760      +/-   ##
   ==========================================
   + Coverage   40.23%   40.28%   +0.04%     
   ==========================================
     Files         455      455              
     Lines       53726    53852     +126     
   ==========================================
   + Hits        21619    21696      +77     
   - Misses      32107    32156      +49     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `43.31% <58.33%> (+1.30%)` | :arrow_up: |
   | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `41.17% <0.00%> (-1.58%)` | :arrow_down: |
   | [sdks/python/apache\_beam/portability/python\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvcHl0aG9uX3VybnMucHk=) | `100.00% <0.00%> (ø)` | |
   | [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `23.36% <0.00%> (ø)` | |
   | [...apache\_beam/portability/api/beam\_runner\_api\_pb2.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjIucHk=) | `100.00% <0.00%> (ø)` | |
   | [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.92% <0.00%> (+0.24%)` | :arrow_up: |
   | [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `43.73% <0.00%> (+0.76%)` | :arrow_up: |
   | [sdks/python/apache\_beam/transforms/core.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb3JlLnB5) | `39.22% <0.00%> (+0.94%)` | :arrow_up: |
   | [...s/python/apache\_beam/testing/synthetic\_pipeline.py](https://codecov.io/gh/apache/beam/pull/12760/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9zeW50aGV0aWNfcGlwZWxpbmUucHk=) | `23.45% <0.00%> (+2.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=footer). Last update [2f8ea18...9dd622c](https://codecov.io/gh/apache/beam/pull/12760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on pull request #12760: [BEAM-10861] Adds URNs and payloads to PubSub transforms to allow runner-native overrides

Posted by GitBox <gi...@apache.org>.

chamikaramj commented on pull request #12760:
URL: https://github.com/apache/beam/pull/12760#issuecomment-691198263


   Ping :)
   
   Tests passed.  PTAL.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org